Nov 03 15:05:28.231223 ip-10-0-139-35 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 15:05:28.231240 ip-10-0-139-35 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 15:05:28.231250 ip-10-0-139-35 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 15:05:28.231612 ip-10-0-139-35 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 15:05:38.451154 ip-10-0-139-35 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 15:05:38.451172 ip-10-0-139-35 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot dadb4ceb9c1b45538b25ad12193082c8 -- Nov 03 15:07:33.142033 ip-10-0-139-35 systemd[1]: Starting Kubernetes Kubelet... Nov 03 15:07:33.564682 ip-10-0-139-35 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 15:07:33.564682 ip-10-0-139-35 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 15:07:33.564682 ip-10-0-139-35 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 15:07:33.564682 ip-10-0-139-35 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 15:07:33.573027 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.572939 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:33.573027 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573022 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:33.573027 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573026 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:33.573027 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573029 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:33.573027 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573032 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573036 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573039 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573042 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573044 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573047 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573050 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573052 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573055 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573057 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573060 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573063 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573065 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573068 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573070 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573073 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573076 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573078 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573081 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573083 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:33.573217 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573086 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573088 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573093 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573097 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573100 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573103 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573105 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573108 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573120 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573122 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573125 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573128 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573131 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573133 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573137 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573139 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573142 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573145 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573147 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:33.573731 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573150 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573152 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573155 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573157 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573160 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573163 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573166 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573168 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573171 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573173 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573177 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573179 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573181 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573184 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573186 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573189 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573193 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573197 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573200 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:33.574179 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573207 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573211 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573213 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573216 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573219 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573221 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573224 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573227 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573230 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573233 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573235 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573238 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573240 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573243 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573245 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573248 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573250 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573253 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573255 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573258 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:33.574658 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573260 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573269 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573272 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573274 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573277 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573279 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573282 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573285 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573287 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573290 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573292 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573295 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573298 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573300 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573303 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573305 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573308 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573310 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573313 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573315 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:33.575145 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573319 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573322 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573324 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.573328 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574182 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574189 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574192 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574195 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574198 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574201 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574204 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574206 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574211 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574215 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574225 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574228 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574231 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574233 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574236 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:33.575617 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574239 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574241 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574244 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574247 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574249 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574252 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574255 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574257 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574260 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574262 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574265 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574268 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574270 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574273 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574276 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574278 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574281 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574283 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574286 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574288 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:33.576092 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574291 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574293 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574296 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574298 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574301 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574304 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574307 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574309 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574317 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574320 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574323 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574325 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574329 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574331 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574334 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574337 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574339 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574342 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574344 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574347 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:33.576561 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574349 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574352 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574354 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574357 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574359 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574362 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574364 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574367 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574370 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574372 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574375 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574377 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574381 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574384 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574387 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574389 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574392 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574394 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574396 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:33.577075 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574399 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574401 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574410 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574413 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574415 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574418 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574421 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574423 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574426 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574428 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574431 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574433 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574436 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574438 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574441 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574443 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574446 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574449 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574452 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574454 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:33.577517 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574457 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574460 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574462 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574465 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574467 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574470 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574472 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574474 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574477 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574479 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574482 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574484 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574969 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574974 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574979 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574981 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574984 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574987 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574990 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574993 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:33.577996 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574995 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.574998 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575000 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575003 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575005 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575008 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575012 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575015 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575018 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575020 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575023 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575025 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575028 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575030 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575033 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575035 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575038 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575040 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575043 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:33.578473 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575047 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575051 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575054 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575057 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575059 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575063 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575066 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575068 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575071 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575074 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575077 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575080 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575082 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575085 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575088 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575090 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575093 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575095 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575097 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575100 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:33.578931 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575102 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575105 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575108 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575111 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575113 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575116 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575118 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575121 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575123 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575126 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575128 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575131 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575133 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575136 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575138 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575141 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575143 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575146 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575149 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:33.579500 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575151 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575154 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575157 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575160 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575163 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575166 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575169 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575171 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575174 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575176 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575178 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575181 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575183 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575186 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575188 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575191 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575193 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575196 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575198 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575200 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:33.579959 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575203 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575206 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575208 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575211 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575213 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575216 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575219 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575221 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575224 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575226 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575229 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575231 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575234 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575236 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575238 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575242 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575244 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575247 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575249 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.575251 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:33.580426 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576134 2575 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576144 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576151 2575 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576157 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576166 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576170 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576174 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576180 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576183 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576186 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576189 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576193 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576196 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576199 2575 flags.go:64] FLAG: --cgroup-root="" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576201 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576204 2575 flags.go:64] FLAG: --client-ca-file="" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576207 2575 flags.go:64] FLAG: --cloud-provider="external" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576210 2575 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576214 2575 flags.go:64] FLAG: --cluster-domain="" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576216 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576219 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576223 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576226 2575 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576230 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 15:07:33.580916 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576234 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576237 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576240 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576243 2575 flags.go:64] FLAG: --contention-profiling="false" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576246 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576249 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576253 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576255 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576260 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576263 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576266 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576268 2575 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576272 2575 flags.go:64] FLAG: --enable-server="true" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576275 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576282 2575 flags.go:64] FLAG: --event-burst="100" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576285 2575 flags.go:64] FLAG: --event-qps="50" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576289 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576292 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576295 2575 flags.go:64] FLAG: --eviction-hard="" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576299 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576302 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576305 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576308 2575 flags.go:64] FLAG: --eviction-soft="" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576311 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576314 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 15:07:33.581464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576317 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576320 2575 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576323 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576329 2575 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576332 2575 flags.go:64] FLAG: --feature-gates="" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576336 2575 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576339 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576342 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576345 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576349 2575 flags.go:64] FLAG: --healthz-port="10248" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576352 2575 flags.go:64] FLAG: --help="false" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576354 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576358 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576361 2575 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576364 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576368 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576371 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576374 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576376 2575 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576379 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576382 2575 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576386 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576389 2575 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576392 2575 flags.go:64] FLAG: --kube-reserved="" Nov 03 15:07:33.582128 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576395 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576398 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576401 2575 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576404 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576407 2575 flags.go:64] FLAG: --lock-file="" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576410 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576413 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576416 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576422 2575 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576424 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576427 2575 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576432 2575 flags.go:64] FLAG: --logging-format="text" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576434 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576438 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576441 2575 flags.go:64] FLAG: --manifest-url="" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576444 2575 flags.go:64] FLAG: --manifest-url-header="" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576448 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576451 2575 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576455 2575 flags.go:64] FLAG: --max-pods="110" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576458 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576461 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576464 2575 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576468 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576470 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576473 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 15:07:33.582691 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576476 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576483 2575 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576486 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576489 2575 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576493 2575 flags.go:64] FLAG: --pod-cidr="" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576496 2575 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576499 2575 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576502 2575 flags.go:64] FLAG: --pods-per-core="0" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576505 2575 flags.go:64] FLAG: --port="10250" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576508 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576511 2575 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0321c9227970242b8" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576514 2575 flags.go:64] FLAG: --qos-reserved="" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576517 2575 flags.go:64] FLAG: --read-only-port="10255" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576520 2575 flags.go:64] FLAG: --register-node="true" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576523 2575 flags.go:64] FLAG: --register-with-taints="" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576527 2575 flags.go:64] FLAG: --registry-burst="10" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576530 2575 flags.go:64] FLAG: --registry-qps="5" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576532 2575 flags.go:64] FLAG: --reserved-cpus="" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576537 2575 flags.go:64] FLAG: --reserved-memory="" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576540 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576543 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576546 2575 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576549 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576552 2575 flags.go:64] FLAG: --runonce="false" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576555 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576558 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576561 2575 flags.go:64] FLAG: --seccomp-default="false" Nov 03 15:07:33.583302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576564 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576567 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576571 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576574 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576577 2575 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576580 2575 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576583 2575 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576586 2575 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576589 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576593 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576596 2575 flags.go:64] FLAG: --system-cgroups="" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576600 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576605 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576607 2575 flags.go:64] FLAG: --tls-cert-file="" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576610 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576614 2575 flags.go:64] FLAG: --tls-min-version="" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576617 2575 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576619 2575 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576622 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576625 2575 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576628 2575 flags.go:64] FLAG: --v="2" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576632 2575 flags.go:64] FLAG: --version="false" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576636 2575 flags.go:64] FLAG: --vmodule="" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576641 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.576644 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 15:07:33.583961 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576785 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576789 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576793 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576795 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576798 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576800 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576804 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576808 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576811 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576814 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576817 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576820 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576822 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576825 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576827 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576830 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576834 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576836 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576839 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576842 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:33.584974 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576844 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576847 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576849 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576852 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576854 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576857 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576859 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576862 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576865 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576867 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576870 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576872 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576874 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576877 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576879 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576882 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576884 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576887 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576889 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:33.585830 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576891 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576894 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576896 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576899 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576903 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576905 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576908 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576910 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576929 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576933 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576937 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576940 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576943 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576945 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576948 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576950 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576955 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576959 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576962 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:33.586613 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576965 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576967 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576970 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576973 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576975 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576977 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576980 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576982 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576985 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576987 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576990 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576992 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576995 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.576997 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577000 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577002 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577005 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577007 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577010 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577013 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:33.587494 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577015 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577018 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577022 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577025 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577027 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577030 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577033 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577035 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577038 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577040 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577043 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577045 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577047 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577050 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577053 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577055 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577057 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577060 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577062 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:33.588355 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577064 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577067 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577069 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577073 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577076 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577078 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577081 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577084 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577086 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.577112 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577147 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577151 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577154 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577157 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577160 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577163 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:33.589209 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577165 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577168 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577170 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577174 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577176 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577179 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577181 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577184 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577186 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577189 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577191 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577194 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577196 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577199 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577201 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577204 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577206 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577209 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577211 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:33.589625 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577215 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577219 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577222 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577225 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577227 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577230 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577233 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577235 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577237 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577240 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577243 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577246 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577248 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577251 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577253 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577256 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577259 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577262 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577264 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577267 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:33.590376 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577269 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577272 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577274 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577277 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577279 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577282 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577284 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577287 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577290 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577292 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577294 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577297 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577299 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577302 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577304 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577307 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577310 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577313 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577315 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:33.591220 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577317 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577320 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577322 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577325 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577328 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577330 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577333 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577336 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577338 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577342 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577346 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577350 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577353 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577355 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577358 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577361 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577363 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577366 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577368 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:33.591958 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577371 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577373 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577376 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577378 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577381 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577383 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577386 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577388 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577391 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577393 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577396 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577398 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577401 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577403 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577406 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577408 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577411 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577413 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577416 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577420 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:33.592404 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577422 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577425 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.577429 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.587105 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.587126 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.587149 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.587157 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587266 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587275 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587280 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587296 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587301 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587306 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587311 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587316 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587320 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587325 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587329 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587333 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:33.592936 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587337 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587342 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587346 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587350 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587355 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587359 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587363 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587368 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587373 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587377 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587381 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587385 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587390 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587394 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587399 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587403 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587407 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587413 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587418 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:33.593435 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587422 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587426 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587431 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587435 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587442 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587448 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587452 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587457 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587461 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587466 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587470 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587474 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587478 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587482 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587487 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587491 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587495 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587499 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587503 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:33.593897 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587507 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587512 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587516 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587520 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587524 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587529 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587534 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587540 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587546 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587552 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587556 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587561 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587566 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587571 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587575 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587579 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587584 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587588 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587593 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:33.594593 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587597 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587601 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587605 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587609 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587614 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587618 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587622 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587626 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587630 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587635 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587639 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587643 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587647 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587651 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587656 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587661 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587665 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587669 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587673 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587678 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:33.595465 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587682 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587686 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587691 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587716 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587721 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587725 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587731 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587735 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587740 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587744 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587748 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587752 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587756 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587761 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587765 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587770 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:33.596133 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.587774 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.587815 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588045 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588053 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588057 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588062 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588066 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588071 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588075 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588079 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588083 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588088 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588092 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588097 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588101 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588105 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:33.596800 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588109 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588113 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588119 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588126 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588131 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588135 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588139 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588145 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588149 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588153 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588157 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588161 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588166 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588170 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588174 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588178 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588182 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588187 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588191 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588196 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:33.597296 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588200 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588205 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588209 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588214 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588218 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588223 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588227 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588232 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588237 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588241 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588245 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588249 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588254 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588258 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588262 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588266 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588270 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588274 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588279 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588283 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:33.597812 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588289 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588293 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588297 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588301 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588305 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588309 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588313 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588317 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588321 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588326 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588330 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588334 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588338 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588342 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588346 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588351 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588354 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588358 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588362 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:33.603998 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588367 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588371 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588377 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588383 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588389 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588393 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588398 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588402 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588406 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588410 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588414 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588418 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588422 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588426 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588431 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588436 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588441 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588445 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588450 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:33.604446 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588454 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588458 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588462 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588466 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588470 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588474 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588478 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588483 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588487 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588492 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588496 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588500 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588504 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588509 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.588546 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588600 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:33.604904 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588606 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588611 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588616 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588622 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588629 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588635 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588640 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588644 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588648 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588652 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588656 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588661 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588666 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588670 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588674 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588678 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588682 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588686 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:33.605274 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588691 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588695 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588717 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588721 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588726 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588730 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588734 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588737 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588741 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588746 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588750 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588754 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588758 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588763 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588767 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588771 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588775 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588779 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588783 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588787 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:33.605721 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588791 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588795 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588800 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588804 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588808 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588812 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588816 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588820 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588824 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588828 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588833 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588837 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588843 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588848 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588852 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588856 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588860 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588864 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588868 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588873 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:33.606205 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588876 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588880 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588885 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588890 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588894 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588898 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588903 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588907 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588911 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588916 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588920 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588924 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588929 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588934 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588938 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588942 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588946 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588950 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588954 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588958 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:33.606765 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588963 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588967 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588971 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588975 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588979 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588983 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588987 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588991 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588995 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.588999 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589003 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589007 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589011 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589015 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589020 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589024 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589028 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589032 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589036 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589040 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:33.607432 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589044 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:33.607990 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589048 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:33.607990 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589052 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:33.607990 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589056 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:33.607990 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589060 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:33.607990 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589066 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:33.607990 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:33.589070 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:33.607990 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.589143 2575 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 15:07:33.607990 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.591695 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 15:07:33.607990 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.592499 2575 server.go:1014] "Starting client certificate rotation" Nov 03 15:07:33.607990 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.592603 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 15:07:33.607990 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.592639 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 15:07:33.607990 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.594617 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 15:07:33.607990 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.601686 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 15:07:33.612539 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.612518 2575 log.go:25] "Validated CRI v1 runtime API" Nov 03 15:07:33.615844 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.615827 2575 log.go:25] "Validated CRI v1 image API" Nov 03 15:07:33.616640 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.616617 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 15:07:33.619941 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.619916 2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8da7b8f2-04e3-469c-bf59-3c0b6013c092:/dev/nvme0n1p4 bf265bef-b8b4-48d9-80f0-c6cb0507c88b:/dev/nvme0n1p3] Nov 03 15:07:33.619992 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.619941 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 15:07:33.624896 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.624782 2575 manager.go:217] Machine: {Timestamp:2026-11-03 15:07:33.623476744 +0000 UTC m=+0.354765649 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090804 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27b9e33916635998834a3aa03e3cc5 SystemUUID:ec27b9e3-3916-6359-9883-4a3aa03e3cc5 BootID:dadb4ceb-9c1b-4553-8b25-ad12193082c8 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6d:0a:70:06:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6d:0a:70:06:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:3e:a5:45:bb:ed Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 15:07:33.624982 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.624897 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 15:07:33.625011 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.624998 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 15:07:33.628104 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.628083 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 15:07:33.628239 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.628105 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-35.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 15:07:33.628288 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.628242 2575 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 15:07:33.628288 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.628250 2575 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 15:07:33.628340 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.628296 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 15:07:33.628340 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.628312 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 15:07:33.629452 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.629442 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 15:07:33.633409 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.633397 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 15:07:33.633529 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.633520 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 15:07:33.634337 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.634326 2575 kubelet.go:493] "Attempting to sync node with API server" Nov 03 15:07:33.634372 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.634342 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 15:07:33.634372 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.634352 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 15:07:33.634372 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.634363 2575 kubelet.go:405] "Adding apiserver pod source" Nov 03 15:07:33.634372 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.634371 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 15:07:33.634683 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.634672 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 15:07:33.636998 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.636981 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 15:07:33.642543 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.642528 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 15:07:33.643422 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.643408 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 15:07:33.643484 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.643448 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 15:07:33.643655 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.643642 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 15:07:33.643733 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.643663 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 15:07:33.643733 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.643673 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 15:07:33.643733 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.643683 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 15:07:33.643733 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.643692 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 15:07:33.643733 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.643720 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 15:07:33.644274 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.644264 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 15:07:33.644325 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.644277 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 15:07:33.644325 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.644298 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 15:07:33.644325 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.644309 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 15:07:33.644425 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.644326 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 15:07:33.644425 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.644341 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 15:07:33.645736 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.645724 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 15:07:33.645798 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.645740 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 15:07:33.649527 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.649511 2575 server.go:1263] "Started kubelet" Nov 03 15:07:33.649628 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.649604 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 15:07:33.653574 ip-10-0-139-35 systemd[1]: Started Kubernetes Kubelet. Nov 03 15:07:33.655528 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.649602 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 15:07:33.655601 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.655551 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 15:07:33.657171 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.657149 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 15:07:33.657850 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.657838 2575 server.go:317] "Adding debug handlers to kubelet server" Nov 03 15:07:33.658791 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:33.658775 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 15:07:33.664624 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.664610 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 15:07:33.665890 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.665855 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 15:07:33.666138 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:33.665958 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-35.us-west-1.compute.internal\" not found" Nov 03 15:07:33.666638 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.666618 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 15:07:33.666920 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.666692 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 15:07:33.666920 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.666727 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 15:07:33.666920 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.666877 2575 reconstruct.go:97] "Volume reconstruction finished" Nov 03 15:07:33.666920 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.666884 2575 reconciler.go:29] "Reconciler: start to sync state" Nov 03 15:07:33.667192 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.667169 2575 factory.go:55] Registering systemd factory Nov 03 15:07:33.667267 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.667196 2575 factory.go:223] Registration of the systemd container factory successfully Nov 03 15:07:33.667447 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.667429 2575 factory.go:153] Registering CRI-O factory Nov 03 15:07:33.667447 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.667447 2575 factory.go:223] Registration of the crio container factory successfully Nov 03 15:07:33.667551 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.667499 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 15:07:33.667551 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.667518 2575 factory.go:103] Registering Raw factory Nov 03 15:07:33.667551 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.667540 2575 manager.go:1199] Started watching for new ooms in manager Nov 03 15:07:33.668586 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.668566 2575 manager.go:319] Starting recovery of all containers Nov 03 15:07:33.678544 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.678528 2575 manager.go:324] Recovery completed Nov 03 15:07:33.683117 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.683102 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:07:33.687012 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.686989 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:07:33.687073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.687021 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:07:33.687073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.687035 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:07:33.687542 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.687527 2575 cpu_manager.go:226] "Starting" policy="none" Nov 03 15:07:33.687620 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.687542 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 15:07:33.687620 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.687561 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 15:07:33.687620 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.687533 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:07:33.688339 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.688325 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:07:33.688393 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.688349 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:07:33.688393 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.688359 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:07:33.689850 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.689837 2575 policy_none.go:50] "Start" Nov 03 15:07:33.689850 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.689852 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 15:07:33.689965 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.689865 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 15:07:33.691789 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.691778 2575 policy_none.go:44] "Start" Nov 03 15:07:33.739362 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.739203 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 15:07:33.743191 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.743177 2575 manager.go:341] "Starting Device Plugin manager" Nov 03 15:07:33.743255 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:33.743217 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 15:07:33.743255 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.743227 2575 server.go:87] "Starting device plugin registration server" Nov 03 15:07:33.743466 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.743454 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 15:07:33.743502 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.743469 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 15:07:33.743587 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.743568 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 15:07:33.743673 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.743661 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 15:07:33.743747 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.743674 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 15:07:33.744189 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:33.744167 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 15:07:33.744280 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:33.744216 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-35.us-west-1.compute.internal\" not found" Nov 03 15:07:33.797943 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.797924 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-35.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 15:07:33.799428 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:33.797931 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-35.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 15:07:33.799428 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:33.797930 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-35.us-west-1.compute.internal.18e4916e18d72921 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-35.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-35.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 15:07:33.649484065 +0000 UTC m=+0.380772969,LastTimestamp:2026-11-03 15:07:33.649484065 +0000 UTC m=+0.380772969,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-35.us-west-1.compute.internal,}" Nov 03 15:07:33.810923 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.810891 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 15:07:33.812240 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.812224 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 15:07:33.812324 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.812248 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 15:07:33.812324 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.812268 2575 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 15:07:33.812324 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:33.812307 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 15:07:33.844566 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.844515 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:07:33.845373 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.845358 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:07:33.845457 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.845392 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:07:33.845457 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.845410 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:07:33.845457 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.845439 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:33.878768 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.878742 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 15:07:33.878864 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.878812 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l6gt8" Nov 03 15:07:33.912653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.912631 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal"] Nov 03 15:07:33.912744 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.912684 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:07:33.913460 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.913446 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:07:33.913522 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.913472 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:07:33.913522 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.913483 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:07:33.915378 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.915366 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:07:33.915528 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.915515 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:33.915563 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.915543 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:07:33.916111 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.916092 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:07:33.916165 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.916138 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:07:33.916165 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.916141 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:07:33.916165 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.916156 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:07:33.916277 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.916169 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:07:33.916277 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.916178 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:07:33.917892 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.917876 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:33.917949 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.917902 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:07:33.918596 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.918579 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:07:33.918653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.918611 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:07:33.918653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.918625 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:07:33.950521 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:33.950504 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-35.us-west-1.compute.internal\" not found" node="ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:33.953393 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:33.953373 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-35.us-west-1.compute.internal\" not found" node="ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:33.958889 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.958875 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:07:33.968931 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.968915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fabcd339c8e2b42bb3c3bff7d173edec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"fabcd339c8e2b42bb3c3bff7d173edec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:33.968999 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.968940 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fabcd339c8e2b42bb3c3bff7d173edec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"fabcd339c8e2b42bb3c3bff7d173edec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:33.968999 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:33.968956 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d955cef443a4983dada78f8e0a328c65-config\") pod \"kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"d955cef443a4983dada78f8e0a328c65\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:34.007601 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.007586 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:07:34.058152 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.058134 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:07:34.059176 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.059159 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:07:34.059470 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.059456 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:34.059791 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:34.059508 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-35.us-west-1.compute.internal\": node \"ip-10-0-139-35.us-west-1.compute.internal\" not found" Nov 03 15:07:34.066124 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.066107 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:34.069993 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.069976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fabcd339c8e2b42bb3c3bff7d173edec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"fabcd339c8e2b42bb3c3bff7d173edec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:34.070066 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.070002 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fabcd339c8e2b42bb3c3bff7d173edec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"fabcd339c8e2b42bb3c3bff7d173edec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:34.070066 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.070021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d955cef443a4983dada78f8e0a328c65-config\") pod \"kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"d955cef443a4983dada78f8e0a328c65\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:34.070134 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.070072 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d955cef443a4983dada78f8e0a328c65-config\") pod \"kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"d955cef443a4983dada78f8e0a328c65\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:34.070134 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.070079 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fabcd339c8e2b42bb3c3bff7d173edec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"fabcd339c8e2b42bb3c3bff7d173edec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:34.070134 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.070087 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fabcd339c8e2b42bb3c3bff7d173edec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"fabcd339c8e2b42bb3c3bff7d173edec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:34.137365 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.137317 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 15:07:34.137431 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.137402 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:34.196627 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.196607 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 15:07:34.253269 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.253249 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:34.256344 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.256321 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:07:34.592510 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.592440 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 15:07:34.593060 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.592554 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 15:07:34.593060 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.592595 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 15:07:34.593060 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.592596 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 15:07:34.593060 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:34.592627 2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a2ce9ff9192244e67a5502d625692119-459a7f00d10b34a9.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-35.us-west-1.compute.internal.18e4916e1b1432d4\": read tcp 10.0.139.35:35094->18.119.150.73:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-35.us-west-1.compute.internal.18e4916e1b1432d4 default 5144 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-35.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-35.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-35.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 15:07:33 +0000 UTC,LastTimestamp:2026-11-03 15:07:33.845415792 +0000 UTC m=+0.576704698,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-35.us-west-1.compute.internal,}" Nov 03 15:07:34.635466 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.635447 2575 apiserver.go:52] "Watching apiserver" Nov 03 15:07:34.665532 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.665512 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 15:07:34.882647 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.882574 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 15:02:34 +0000 UTC" deadline="2028-08-11 09:15:16.925232612 +0000 UTC" Nov 03 15:07:34.882647 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.882610 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15522h7m42.042625779s" Nov 03 15:07:34.907508 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.907483 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 15:07:34.907831 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.907811 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hwchb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7","openshift-image-registry/node-ca-7fz2g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-rq6pc","openshift-ovn-kubernetes/ovnkube-node-z7xfz","kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal","openshift-dns/node-resolver-cmv4d","openshift-network-diagnostics/network-check-target-qwzv7","kube-system/konnectivity-agent-q5rxt","openshift-cluster-node-tuning-operator/tuned-qx6fv","openshift-multus/multus-additional-cni-plugins-zkw2x","openshift-multus/multus-dtkj5","openshift-multus/network-metrics-daemon-fkx68"] Nov 03 15:07:34.910059 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.910037 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:34.911924 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.911909 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:34.913769 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.913753 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:07:34.915774 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.915758 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:34.915854 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:34.915816 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:07:34.917764 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.917752 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:07:34.919929 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.919917 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.922001 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.921986 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.924107 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.924092 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:07:34.926169 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.926151 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:34.926254 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:34.926216 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:07:34.928279 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.928262 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:07:34.930371 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.930356 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:34.932362 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.932344 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:34.934346 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:34.932646 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:07:34.963641 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.963617 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 15:07:34.963767 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.963729 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 15:07:34.963767 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.963744 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 15:07:34.963882 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.963822 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 15:07:34.963882 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.963847 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 15:07:34.963882 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.963864 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qsch2\"" Nov 03 15:07:34.964021 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.963964 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 15:07:34.964496 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.964478 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 15:07:34.964587 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.964573 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fsh2r\"" Nov 03 15:07:34.964764 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.964753 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 15:07:34.966311 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.966297 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 15:07:34.966482 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.966468 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jpxkd\"" Nov 03 15:07:34.966542 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.966527 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 15:07:34.966618 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.966604 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 15:07:34.968252 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.968239 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 15:07:34.970373 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.970359 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 15:07:34.970504 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.970488 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 15:07:34.970553 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.970509 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 15:07:34.970598 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.970587 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6cpt6\"" Nov 03 15:07:34.975807 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.975791 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-log-socket\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.975892 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.975814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d5de266-de42-430c-b364-d12951764b40-iptables-alerter-script\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:07:34.975892 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.975830 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-etc-selinux\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:34.975892 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.975843 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-system-cni-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.975892 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.975858 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-k8s-cni-cncf-io\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.976050 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.975899 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-host\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:34.976050 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.975942 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1129e2e-263b-48b8-a239-97d9164beca1-tmp\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:34.976050 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.975974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:34.976050 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976004 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4f18ec9-6d01-4a3d-9845-58b607a6225f-agent-certs\") pod \"konnectivity-agent-q5rxt\" (UID: \"c4f18ec9-6d01-4a3d-9845-58b607a6225f\") " pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:07:34.976050 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976020 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:34.976050 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-run\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:34.976292 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976057 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-var-lib-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.976292 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976083 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-run-ovn-kubernetes\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.976292 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976101 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m4st\" (UniqueName: \"kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st\") pod \"network-check-target-qwzv7\" (UID: \"9d58285c-7e8d-4833-87a2-66ba8f398c8d\") " pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:34.976292 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976120 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-slash\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.976292 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976134 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-socket-dir-parent\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.976292 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-netns\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.976292 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-conf-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.976292 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976190 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-multus-certs\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.976292 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976219 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1129e2e-263b-48b8-a239-97d9164beca1-etc-tuned\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:34.976292 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976243 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-cni-netd\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.976292 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976282 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-kubelet\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.976634 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976308 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f997ae6c-4cdd-4def-ac45-e81c30896420-dbus\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:34.976634 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-socket-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:34.976634 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976374 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-run-netns\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.976634 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.976634 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6zql\" (UniqueName: \"kubernetes.io/projected/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-kube-api-access-p6zql\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.976634 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-daemon-config\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.976634 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976499 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-etc-kubernetes\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.976634 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976544 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwhk7\" (UniqueName: \"kubernetes.io/projected/ab771653-7ecd-4022-900f-c02f580bbb01-kube-api-access-bwhk7\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:07:34.976634 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976574 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovnkube-config\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.976634 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976597 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-cni-bin\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgnz5\" (UniqueName: \"kubernetes.io/projected/c46a1f0f-4dc2-4b03-8269-d19a93977b56-kube-api-access-tgnz5\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976685 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/588c75cb-ff10-45c1-be72-625f6d9b9d9e-tmp-dir\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976720 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab771653-7ecd-4022-900f-c02f580bbb01-host\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976734 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8szqq\" (UniqueName: \"kubernetes.io/projected/a6fc4400-11f3-42a8-afa3-68143e29eb10-kube-api-access-8szqq\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976751 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-lib-modules\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976766 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976780 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-registration-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976793 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-ovn\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-env-overrides\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976880 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-kubernetes\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976904 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976929 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-device-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976966 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-sys-fs\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.976992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdj7s\" (UniqueName: \"kubernetes.io/projected/b304a74d-882a-44e8-a696-4aff7665157c-kube-api-access-vdj7s\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977016 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-etc-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.977073 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977039 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-node-log\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab771653-7ecd-4022-900f-c02f580bbb01-serviceca\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977101 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-systemd\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7pzg\" (UniqueName: \"kubernetes.io/projected/a1129e2e-263b-48b8-a239-97d9164beca1-kube-api-access-g7pzg\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977153 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d5de266-de42-430c-b364-d12951764b40-host-slash\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-modprobe-d\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f997ae6c-4cdd-4def-ac45-e81c30896420-kubelet-config\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977194 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977212 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xpq6\" (UniqueName: \"kubernetes.io/projected/5c8010d3-c54b-48b1-b792-fbd8933863a2-kube-api-access-5xpq6\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-cni-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-cnibin\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977301 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/588c75cb-ff10-45c1-be72-625f6d9b9d9e-hosts-file\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5ct\" (UniqueName: \"kubernetes.io/projected/588c75cb-ff10-45c1-be72-625f6d9b9d9e-kube-api-access-dq5ct\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-system-cni-dir\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-cnibin\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:34.977653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovn-node-metrics-cert\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977401 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977416 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovnkube-script-lib\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977433 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-kubelet\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-systemd\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-cni-bin\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977541 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-hostroot\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977566 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfpfv\" (UniqueName: \"kubernetes.io/projected/7d5de266-de42-430c-b364-d12951764b40-kube-api-access-xfpfv\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977585 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysctl-d\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977606 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysctl-conf\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-cni-multus\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977636 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4f18ec9-6d01-4a3d-9845-58b607a6225f-konnectivity-ca\") pod \"konnectivity-agent-q5rxt\" (UID: \"c4f18ec9-6d01-4a3d-9845-58b607a6225f\") " pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977653 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysconfig\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977672 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-sys\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-var-lib-kubelet\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:34.978126 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-systemd-units\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:34.978582 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:34.977756 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c46a1f0f-4dc2-4b03-8269-d19a93977b56-cni-binary-copy\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.060883 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.060864 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 15:07:35.060981 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.060914 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s65rq" Nov 03 15:07:35.064029 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.064013 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 15:07:35.064246 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.064214 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 15:07:35.064406 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.064394 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 15:07:35.064487 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.064472 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 15:07:35.064543 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.064489 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 15:07:35.064543 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.064493 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dfz9p\"" Nov 03 15:07:35.064543 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.064509 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kdt56\"" Nov 03 15:07:35.064543 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.064536 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 15:07:35.064543 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.064538 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 15:07:35.064745 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.064545 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 15:07:35.064745 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.064542 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cjtnz\"" Nov 03 15:07:35.064745 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.064571 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 15:07:35.064975 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.064961 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cblvv\"" Nov 03 15:07:35.065006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.064972 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 15:07:35.077039 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.077017 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 15:07:35.077039 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.077029 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 15:07:35.077145 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.077063 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 15:07:35.077145 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.077076 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4wckw\"" Nov 03 15:07:35.078297 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-device-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:35.078347 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078305 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-sys-fs\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:35.078347 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdj7s\" (UniqueName: \"kubernetes.io/projected/b304a74d-882a-44e8-a696-4aff7665157c-kube-api-access-vdj7s\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:35.078347 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078336 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-etc-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.078450 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078350 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-node-log\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.078450 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078369 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab771653-7ecd-4022-900f-c02f580bbb01-serviceca\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:07:35.078450 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-systemd\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.078450 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078393 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-sys-fs\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:35.078450 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7pzg\" (UniqueName: \"kubernetes.io/projected/a1129e2e-263b-48b8-a239-97d9164beca1-kube-api-access-g7pzg\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.078450 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078396 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-device-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:35.078450 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078418 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-etc-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.078450 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078447 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-node-log\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.078781 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078512 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d5de266-de42-430c-b364-d12951764b40-host-slash\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:07:35.078781 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-modprobe-d\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.078781 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078557 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-systemd\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.078781 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f997ae6c-4cdd-4def-ac45-e81c30896420-kubelet-config\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:35.078781 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078564 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d5de266-de42-430c-b364-d12951764b40-host-slash\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:07:35.078781 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:35.078781 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:35.078781 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078658 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-modprobe-d\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.078781 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078639 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f997ae6c-4cdd-4def-ac45-e81c30896420-kubelet-config\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:35.078781 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xpq6\" (UniqueName: \"kubernetes.io/projected/5c8010d3-c54b-48b1-b792-fbd8933863a2-kube-api-access-5xpq6\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:35.078781 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-cni-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.078781 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.078733 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:35.078781 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078745 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-cnibin\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.078781 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078768 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/588c75cb-ff10-45c1-be72-625f6d9b9d9e-hosts-file\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.078797 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret podName:f997ae6c-4cdd-4def-ac45-e81c30896420 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:35.578778565 +0000 UTC m=+2.310067472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret") pod "global-pull-secret-syncer-hwchb" (UID: "f997ae6c-4cdd-4def-ac45-e81c30896420") : object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078807 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab771653-7ecd-4022-900f-c02f580bbb01-serviceca\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078823 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/588c75cb-ff10-45c1-be72-625f6d9b9d9e-hosts-file\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078817 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5ct\" (UniqueName: \"kubernetes.io/projected/588c75cb-ff10-45c1-be72-625f6d9b9d9e-kube-api-access-dq5ct\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078870 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-cnibin\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-system-cni-dir\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078903 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-cni-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078906 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-cnibin\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078935 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-system-cni-dir\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078939 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-cnibin\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovn-node-metrics-cert\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.078972 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079002 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovnkube-script-lib\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079016 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-kubelet\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-systemd\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.079419 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079067 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-cni-bin\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079086 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-systemd\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079089 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-hostroot\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079109 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-hostroot\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079067 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-kubelet\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079121 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfpfv\" (UniqueName: \"kubernetes.io/projected/7d5de266-de42-430c-b364-d12951764b40-kube-api-access-xfpfv\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079097 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-cni-bin\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysctl-d\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079179 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysctl-conf\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079242 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysctl-d\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079262 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-cni-multus\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079253 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079294 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4f18ec9-6d01-4a3d-9845-58b607a6225f-konnectivity-ca\") pod \"konnectivity-agent-q5rxt\" (UID: \"c4f18ec9-6d01-4a3d-9845-58b607a6225f\") " pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079313 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysconfig\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079307 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-cni-multus\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-sys\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-var-lib-kubelet\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.080090 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079395 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-systemd-units\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079404 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysctl-conf\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c46a1f0f-4dc2-4b03-8269-d19a93977b56-cni-binary-copy\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079451 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-log-socket\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079464 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079475 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d5de266-de42-430c-b364-d12951764b40-iptables-alerter-script\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079422 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-sys\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079475 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysconfig\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079512 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-var-lib-kubelet\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079529 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-log-socket\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079561 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovnkube-script-lib\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079570 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-systemd-units\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079583 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-etc-selinux\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079513 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-etc-selinux\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-system-cni-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-k8s-cni-cncf-io\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-host\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.081006 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079683 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1129e2e-263b-48b8-a239-97d9164beca1-tmp\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079760 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-k8s-cni-cncf-io\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4f18ec9-6d01-4a3d-9845-58b607a6225f-agent-certs\") pod \"konnectivity-agent-q5rxt\" (UID: \"c4f18ec9-6d01-4a3d-9845-58b607a6225f\") " pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079826 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079852 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-run\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-var-lib-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079885 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4f18ec9-6d01-4a3d-9845-58b607a6225f-konnectivity-ca\") pod \"konnectivity-agent-q5rxt\" (UID: \"c4f18ec9-6d01-4a3d-9845-58b607a6225f\") " pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079903 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-run-ovn-kubernetes\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4st\" (UniqueName: \"kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st\") pod \"network-check-target-qwzv7\" (UID: \"9d58285c-7e8d-4833-87a2-66ba8f398c8d\") " pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079952 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-slash\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079974 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-socket-dir-parent\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.079995 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-netns\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-conf-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080042 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-multus-certs\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080112 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-var-lib-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080126 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-run\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080147 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1129e2e-263b-48b8-a239-97d9164beca1-etc-tuned\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.081847 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080165 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-system-cni-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080174 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-cni-netd\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080112 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d5de266-de42-430c-b364-d12951764b40-iptables-alerter-script\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080150 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-multus-certs\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080213 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-host\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080227 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-conf-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080227 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-slash\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080261 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-socket-dir-parent\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080262 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-netns\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-kubelet\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080304 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-run-ovn-kubernetes\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080330 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-cni-netd\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f997ae6c-4cdd-4def-ac45-e81c30896420-dbus\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.080379 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080380 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-socket-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.080462 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs podName:a6fc4400-11f3-42a8-afa3-68143e29eb10 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:35.580439796 +0000 UTC m=+2.311728694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs") pod "network-metrics-daemon-fkx68" (UID: "a6fc4400-11f3-42a8-afa3-68143e29eb10") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080471 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-socket-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:35.082670 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080487 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-kubelet\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.083181 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080508 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f997ae6c-4cdd-4def-ac45-e81c30896420-dbus\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:35.083181 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.082889 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:35.083181 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.083092 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1129e2e-263b-48b8-a239-97d9164beca1-tmp\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.083266 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.083213 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c46a1f0f-4dc2-4b03-8269-d19a93977b56-cni-binary-copy\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.083401 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.083376 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1129e2e-263b-48b8-a239-97d9164beca1-etc-tuned\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.083648 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.083628 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovn-node-metrics-cert\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.083718 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.083659 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4f18ec9-6d01-4a3d-9845-58b607a6225f-agent-certs\") pod \"konnectivity-agent-q5rxt\" (UID: \"c4f18ec9-6d01-4a3d-9845-58b607a6225f\") " pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:07:35.083776 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.080511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-run-netns\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.083776 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.083752 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-run-netns\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.083776 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.083764 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.083908 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.083800 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6zql\" (UniqueName: \"kubernetes.io/projected/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-kube-api-access-p6zql\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.083908 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.083823 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.083908 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.083832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-daemon-config\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.083908 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.083873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-etc-kubernetes\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.083908 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.083905 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwhk7\" (UniqueName: \"kubernetes.io/projected/ab771653-7ecd-4022-900f-c02f580bbb01-kube-api-access-bwhk7\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:07:35.084309 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.084289 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-daemon-config\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.085828 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.085805 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-etc-kubernetes\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.085981 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.085964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovnkube-config\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.086055 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-cni-bin\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.086055 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgnz5\" (UniqueName: \"kubernetes.io/projected/c46a1f0f-4dc2-4b03-8269-d19a93977b56-kube-api-access-tgnz5\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.086151 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/588c75cb-ff10-45c1-be72-625f6d9b9d9e-tmp-dir\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:07:35.086151 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086100 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab771653-7ecd-4022-900f-c02f580bbb01-host\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:07:35.086151 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086131 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-cni-bin\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.086294 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8szqq\" (UniqueName: \"kubernetes.io/projected/a6fc4400-11f3-42a8-afa3-68143e29eb10-kube-api-access-8szqq\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:35.086294 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086252 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-lib-modules\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.086388 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086254 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab771653-7ecd-4022-900f-c02f580bbb01-host\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:07:35.086388 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:35.086388 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-registration-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:35.086532 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086386 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-lib-modules\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.086532 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-ovn\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.086532 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086451 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-env-overrides\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.086532 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086475 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/588c75cb-ff10-45c1-be72-625f6d9b9d9e-tmp-dir\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:07:35.086532 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086482 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-kubernetes\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.086532 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086501 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-ovn\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.086532 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086482 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-registration-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:35.087163 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086477 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovnkube-config\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.087163 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086881 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-kubernetes\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.087163 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.086893 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:35.087163 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.087067 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:35.087163 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.087132 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:35.087336 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.087232 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-env-overrides\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.138742 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.138667 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 15:07:35.138742 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.138693 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 15:07:35.138742 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.138724 2575 projected.go:196] Error preparing data for projected volume kube-api-access-9m4st for pod openshift-network-diagnostics/network-check-target-qwzv7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:35.138950 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.138821 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st podName:9d58285c-7e8d-4833-87a2-66ba8f398c8d nodeName:}" failed. No retries permitted until 2026-11-03 15:07:35.638804094 +0000 UTC m=+2.370093025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9m4st" (UniqueName: "kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st") pod "network-check-target-qwzv7" (UID: "9d58285c-7e8d-4833-87a2-66ba8f398c8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:35.140137 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.140114 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdj7s\" (UniqueName: \"kubernetes.io/projected/b304a74d-882a-44e8-a696-4aff7665157c-kube-api-access-vdj7s\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:35.140257 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.140237 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5ct\" (UniqueName: \"kubernetes.io/projected/588c75cb-ff10-45c1-be72-625f6d9b9d9e-kube-api-access-dq5ct\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:07:35.140773 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.140754 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7pzg\" (UniqueName: \"kubernetes.io/projected/a1129e2e-263b-48b8-a239-97d9164beca1-kube-api-access-g7pzg\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.140852 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.140837 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfpfv\" (UniqueName: \"kubernetes.io/projected/7d5de266-de42-430c-b364-d12951764b40-kube-api-access-xfpfv\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:07:35.140895 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.140847 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xpq6\" (UniqueName: \"kubernetes.io/projected/5c8010d3-c54b-48b1-b792-fbd8933863a2-kube-api-access-5xpq6\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:35.147392 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.147368 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgnz5\" (UniqueName: \"kubernetes.io/projected/c46a1f0f-4dc2-4b03-8269-d19a93977b56-kube-api-access-tgnz5\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.147523 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.147507 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwhk7\" (UniqueName: \"kubernetes.io/projected/ab771653-7ecd-4022-900f-c02f580bbb01-kube-api-access-bwhk7\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:07:35.147586 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.147572 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6zql\" (UniqueName: \"kubernetes.io/projected/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-kube-api-access-p6zql\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.147995 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.147977 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8szqq\" (UniqueName: \"kubernetes.io/projected/a6fc4400-11f3-42a8-afa3-68143e29eb10-kube-api-access-8szqq\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:35.250917 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.250884 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:07:35.260459 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.260438 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:07:35.275513 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.275491 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:07:35.281426 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.281401 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:07:35.306628 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.306609 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:35.315234 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.315215 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dtkj5" Nov 03 15:07:35.328526 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.328510 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:07:35.342068 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.342047 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:07:35.346205 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.346192 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:07:35.368066 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:35.368040 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f18ec9_6d01_4a3d_9845_58b607a6225f.slice/crio-ba30a154663ef205fe454179476cf3a18853919cd8be2da60aa2efe644c29903 WatchSource:0}: Error finding container ba30a154663ef205fe454179476cf3a18853919cd8be2da60aa2efe644c29903: Status 404 returned error can't find the container with id ba30a154663ef205fe454179476cf3a18853919cd8be2da60aa2efe644c29903 Nov 03 15:07:35.368607 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:35.368594 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd955cef443a4983dada78f8e0a328c65.slice/crio-63a79073cf5f69c69bc4d1d8f29085e892cbe545d9f536ec9217b224fe6aafcb WatchSource:0}: Error finding container 63a79073cf5f69c69bc4d1d8f29085e892cbe545d9f536ec9217b224fe6aafcb: Status 404 returned error can't find the container with id 63a79073cf5f69c69bc4d1d8f29085e892cbe545d9f536ec9217b224fe6aafcb Nov 03 15:07:35.369396 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:35.369377 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1129e2e_263b_48b8_a239_97d9164beca1.slice/crio-67d34ccb66fedd3d3c7772fe016f5d7e5ea6607d2ec11d416674e3b92ea3842c WatchSource:0}: Error finding container 67d34ccb66fedd3d3c7772fe016f5d7e5ea6607d2ec11d416674e3b92ea3842c: Status 404 returned error can't find the container with id 67d34ccb66fedd3d3c7772fe016f5d7e5ea6607d2ec11d416674e3b92ea3842c Nov 03 15:07:35.369890 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:35.369869 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabcd339c8e2b42bb3c3bff7d173edec.slice/crio-6f735fd43670028f25e285734a6ac5bd76a2753e3c11ab0687aa1864def51645 WatchSource:0}: Error finding container 6f735fd43670028f25e285734a6ac5bd76a2753e3c11ab0687aa1864def51645: Status 404 returned error can't find the container with id 6f735fd43670028f25e285734a6ac5bd76a2753e3c11ab0687aa1864def51645 Nov 03 15:07:35.373014 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.372972 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:07:35.529965 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.529941 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:07:35.591422 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.591396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:35.591531 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.591448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:35.591573 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.591535 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:35.591573 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.591536 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:35.591635 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.591582 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret podName:f997ae6c-4cdd-4def-ac45-e81c30896420 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:36.591569408 +0000 UTC m=+3.322858301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret") pod "global-pull-secret-syncer-hwchb" (UID: "f997ae6c-4cdd-4def-ac45-e81c30896420") : object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:35.591635 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.591595 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs podName:a6fc4400-11f3-42a8-afa3-68143e29eb10 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:36.591588702 +0000 UTC m=+3.322877595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs") pod "network-metrics-daemon-fkx68" (UID: "a6fc4400-11f3-42a8-afa3-68143e29eb10") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:35.655287 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.655251 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:07:35.693021 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.692982 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4st\" (UniqueName: \"kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st\") pod \"network-check-target-qwzv7\" (UID: \"9d58285c-7e8d-4833-87a2-66ba8f398c8d\") " pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:35.693142 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.693126 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 15:07:35.693188 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.693150 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 15:07:35.693188 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.693159 2575 projected.go:196] Error preparing data for projected volume kube-api-access-9m4st for pod openshift-network-diagnostics/network-check-target-qwzv7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:35.693252 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:35.693202 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st podName:9d58285c-7e8d-4833-87a2-66ba8f398c8d nodeName:}" failed. No retries permitted until 2026-11-03 15:07:36.693189975 +0000 UTC m=+3.424478869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m4st" (UniqueName: "kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st") pod "network-check-target-qwzv7" (UID: "9d58285c-7e8d-4833-87a2-66ba8f398c8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:35.816750 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.816663 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" event={"ID":"a1129e2e-263b-48b8-a239-97d9164beca1","Type":"ContainerStarted","Data":"67d34ccb66fedd3d3c7772fe016f5d7e5ea6607d2ec11d416674e3b92ea3842c"} Nov 03 15:07:35.817652 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.817631 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q5rxt" event={"ID":"c4f18ec9-6d01-4a3d-9845-58b607a6225f","Type":"ContainerStarted","Data":"ba30a154663ef205fe454179476cf3a18853919cd8be2da60aa2efe644c29903"} Nov 03 15:07:35.818684 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.818665 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" event={"ID":"fabcd339c8e2b42bb3c3bff7d173edec","Type":"ContainerStarted","Data":"6f735fd43670028f25e285734a6ac5bd76a2753e3c11ab0687aa1864def51645"} Nov 03 15:07:35.819767 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:35.819749 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" event={"ID":"d955cef443a4983dada78f8e0a328c65","Type":"ContainerStarted","Data":"63a79073cf5f69c69bc4d1d8f29085e892cbe545d9f536ec9217b224fe6aafcb"} Nov 03 15:07:36.064302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.064270 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:02:35 +0000 UTC" deadline="2028-07-06 05:43:21.46648475 +0000 UTC" Nov 03 15:07:36.064302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.064295 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14654h35m45.402192688s" Nov 03 15:07:36.113682 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.113613 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:07:36.158485 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:36.158452 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb304a74d_882a_44e8_a696_4aff7665157c.slice/crio-eb974c8097603b0176a5da2b7104a65434f16cd1daf03f37c58fd18b7644b6f0 WatchSource:0}: Error finding container eb974c8097603b0176a5da2b7104a65434f16cd1daf03f37c58fd18b7644b6f0: Status 404 returned error can't find the container with id eb974c8097603b0176a5da2b7104a65434f16cd1daf03f37c58fd18b7644b6f0 Nov 03 15:07:36.162207 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:36.162177 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5de266_de42_430c_b364_d12951764b40.slice/crio-c4baa7e38dc58d63830d401dc214d8a46d3d9ace1005eb075ba9d4483631926b WatchSource:0}: Error finding container c4baa7e38dc58d63830d401dc214d8a46d3d9ace1005eb075ba9d4483631926b: Status 404 returned error can't find the container with id c4baa7e38dc58d63830d401dc214d8a46d3d9ace1005eb075ba9d4483631926b Nov 03 15:07:36.162968 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:36.162946 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8010d3_c54b_48b1_b792_fbd8933863a2.slice/crio-e84d1b045ea2891383e88467e2f57f1b375d351d948f188df591e46e31c27799 WatchSource:0}: Error finding container e84d1b045ea2891383e88467e2f57f1b375d351d948f188df591e46e31c27799: Status 404 returned error can't find the container with id e84d1b045ea2891383e88467e2f57f1b375d351d948f188df591e46e31c27799 Nov 03 15:07:36.199759 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:36.199718 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588c75cb_ff10_45c1_be72_625f6d9b9d9e.slice/crio-3e6b796d62ff9620023eec074e3548c07fc17e74f482e89cfef2cd50af363e29 WatchSource:0}: Error finding container 3e6b796d62ff9620023eec074e3548c07fc17e74f482e89cfef2cd50af363e29: Status 404 returned error can't find the container with id 3e6b796d62ff9620023eec074e3548c07fc17e74f482e89cfef2cd50af363e29 Nov 03 15:07:36.222163 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:36.222141 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46a1f0f_4dc2_4b03_8269_d19a93977b56.slice/crio-fc55f4153c09d413dbf721ad9a66592d3147096497da3502bf7728d943bc3415 WatchSource:0}: Error finding container fc55f4153c09d413dbf721ad9a66592d3147096497da3502bf7728d943bc3415: Status 404 returned error can't find the container with id fc55f4153c09d413dbf721ad9a66592d3147096497da3502bf7728d943bc3415 Nov 03 15:07:36.278071 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:36.278048 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc1fa7c_1f46_4151_9375_cedcec2e8cfa.slice/crio-2203c5ec29ce08b35beabfed77b82c8f6a43e3ea6ef95f99a664fe286b6e7a7f WatchSource:0}: Error finding container 2203c5ec29ce08b35beabfed77b82c8f6a43e3ea6ef95f99a664fe286b6e7a7f: Status 404 returned error can't find the container with id 2203c5ec29ce08b35beabfed77b82c8f6a43e3ea6ef95f99a664fe286b6e7a7f Nov 03 15:07:36.297165 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:07:36.297136 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab771653_7ecd_4022_900f_c02f580bbb01.slice/crio-57ee6785f2343b55754b9bb01099fcfcc9ce3178cde783c74cbc79c9349d31e2 WatchSource:0}: Error finding container 57ee6785f2343b55754b9bb01099fcfcc9ce3178cde783c74cbc79c9349d31e2: Status 404 returned error can't find the container with id 57ee6785f2343b55754b9bb01099fcfcc9ce3178cde783c74cbc79c9349d31e2 Nov 03 15:07:36.600131 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.600046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:36.600131 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.600125 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:36.600394 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:36.600304 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:36.600394 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:36.600356 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs podName:a6fc4400-11f3-42a8-afa3-68143e29eb10 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:38.600339672 +0000 UTC m=+5.331628565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs") pod "network-metrics-daemon-fkx68" (UID: "a6fc4400-11f3-42a8-afa3-68143e29eb10") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:36.600554 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:36.600410 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:36.600554 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:36.600452 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret podName:f997ae6c-4cdd-4def-ac45-e81c30896420 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:38.600444156 +0000 UTC m=+5.331733049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret") pod "global-pull-secret-syncer-hwchb" (UID: "f997ae6c-4cdd-4def-ac45-e81c30896420") : object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:36.700464 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.700431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4st\" (UniqueName: \"kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st\") pod \"network-check-target-qwzv7\" (UID: \"9d58285c-7e8d-4833-87a2-66ba8f398c8d\") " pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:36.701368 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:36.700618 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 15:07:36.701368 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:36.700640 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 15:07:36.701368 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:36.700654 2575 projected.go:196] Error preparing data for projected volume kube-api-access-9m4st for pod openshift-network-diagnostics/network-check-target-qwzv7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:36.701368 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:36.700743 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st podName:9d58285c-7e8d-4833-87a2-66ba8f398c8d nodeName:}" failed. No retries permitted until 2026-11-03 15:07:38.700702865 +0000 UTC m=+5.431991767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m4st" (UniqueName: "kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st") pod "network-check-target-qwzv7" (UID: "9d58285c-7e8d-4833-87a2-66ba8f398c8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:36.813541 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.813046 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:36.813541 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:36.813189 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:07:36.813541 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.813276 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:36.813541 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:36.813342 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:07:36.813541 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.813410 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:36.813541 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:36.813498 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:07:36.823693 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.823659 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rq6pc" event={"ID":"7d5de266-de42-430c-b364-d12951764b40","Type":"ContainerStarted","Data":"c4baa7e38dc58d63830d401dc214d8a46d3d9ace1005eb075ba9d4483631926b"} Nov 03 15:07:36.825795 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.825726 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" event={"ID":"b304a74d-882a-44e8-a696-4aff7665157c","Type":"ContainerStarted","Data":"eb974c8097603b0176a5da2b7104a65434f16cd1daf03f37c58fd18b7644b6f0"} Nov 03 15:07:36.828065 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.828037 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dtkj5" event={"ID":"c46a1f0f-4dc2-4b03-8269-d19a93977b56","Type":"ContainerStarted","Data":"fc55f4153c09d413dbf721ad9a66592d3147096497da3502bf7728d943bc3415"} Nov 03 15:07:36.829993 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.829966 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cmv4d" event={"ID":"588c75cb-ff10-45c1-be72-625f6d9b9d9e","Type":"ContainerStarted","Data":"3e6b796d62ff9620023eec074e3548c07fc17e74f482e89cfef2cd50af363e29"} Nov 03 15:07:36.831466 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.831439 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7fz2g" event={"ID":"ab771653-7ecd-4022-900f-c02f580bbb01","Type":"ContainerStarted","Data":"57ee6785f2343b55754b9bb01099fcfcc9ce3178cde783c74cbc79c9349d31e2"} Nov 03 15:07:36.833633 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.833606 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" event={"ID":"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa","Type":"ContainerStarted","Data":"2203c5ec29ce08b35beabfed77b82c8f6a43e3ea6ef95f99a664fe286b6e7a7f"} Nov 03 15:07:36.836323 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:36.836296 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkw2x" event={"ID":"5c8010d3-c54b-48b1-b792-fbd8933863a2","Type":"ContainerStarted","Data":"e84d1b045ea2891383e88467e2f57f1b375d351d948f188df591e46e31c27799"} Nov 03 15:07:37.065259 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:37.065137 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:02:35 +0000 UTC" deadline="2028-07-14 10:46:03.66079251 +0000 UTC" Nov 03 15:07:37.065259 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:37.065203 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14851h38m26.595593747s" Nov 03 15:07:38.619720 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:38.619258 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:38.619720 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:38.619373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:38.619720 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:38.619516 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:38.619720 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:38.619578 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret podName:f997ae6c-4cdd-4def-ac45-e81c30896420 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:42.619559814 +0000 UTC m=+9.350848732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret") pod "global-pull-secret-syncer-hwchb" (UID: "f997ae6c-4cdd-4def-ac45-e81c30896420") : object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:38.619720 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:38.619653 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:38.619720 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:38.619695 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs podName:a6fc4400-11f3-42a8-afa3-68143e29eb10 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:42.619680801 +0000 UTC m=+9.350969697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs") pod "network-metrics-daemon-fkx68" (UID: "a6fc4400-11f3-42a8-afa3-68143e29eb10") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:38.721532 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:38.721196 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4st\" (UniqueName: \"kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st\") pod \"network-check-target-qwzv7\" (UID: \"9d58285c-7e8d-4833-87a2-66ba8f398c8d\") " pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:38.721532 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:38.721388 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 15:07:38.721532 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:38.721414 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 15:07:38.721532 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:38.721428 2575 projected.go:196] Error preparing data for projected volume kube-api-access-9m4st for pod openshift-network-diagnostics/network-check-target-qwzv7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:38.721532 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:38.721491 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st podName:9d58285c-7e8d-4833-87a2-66ba8f398c8d nodeName:}" failed. No retries permitted until 2026-11-03 15:07:42.721472831 +0000 UTC m=+9.452761733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m4st" (UniqueName: "kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st") pod "network-check-target-qwzv7" (UID: "9d58285c-7e8d-4833-87a2-66ba8f398c8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:38.813542 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:38.813038 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:38.813542 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:38.813201 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:07:38.813542 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:38.813269 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:38.813542 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:38.813349 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:07:38.813542 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:38.813400 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:38.813542 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:38.813469 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:07:40.814421 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:40.813734 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:40.814421 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:40.814067 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:07:40.814421 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:40.814163 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:40.814421 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:40.814249 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:07:40.814421 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:40.814289 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:40.814421 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:40.814347 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:07:40.873197 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:40.873158 2575 generic.go:356] "Generic (PLEG): container finished" podID="5c8010d3-c54b-48b1-b792-fbd8933863a2" containerID="853dbc9869c3fba680c4b1bb079a4823316022cc2adccfdcd5a73e8d018d30f6" exitCode=0 Nov 03 15:07:40.873373 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:40.873251 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkw2x" event={"ID":"5c8010d3-c54b-48b1-b792-fbd8933863a2","Type":"ContainerDied","Data":"853dbc9869c3fba680c4b1bb079a4823316022cc2adccfdcd5a73e8d018d30f6"} Nov 03 15:07:40.879146 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:40.879114 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" event={"ID":"d955cef443a4983dada78f8e0a328c65","Type":"ContainerStarted","Data":"5f799f874ac83e1fa6521ec1dd628a31a0bd9d1a7c8a184670798a33cf1ac513"} Nov 03 15:07:40.883653 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:40.883626 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q5rxt" event={"ID":"c4f18ec9-6d01-4a3d-9845-58b607a6225f","Type":"ContainerStarted","Data":"4a38ceb4fd0187560fcfab7693272fec740038443a8c7458fd853a9d0e160870"} Nov 03 15:07:40.887302 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:40.887275 2575 generic.go:356] "Generic (PLEG): container finished" podID="fabcd339c8e2b42bb3c3bff7d173edec" containerID="cc03d66fb51322da4a1ff0f26ca5dc4080274054bd2a01a2289c9e445827ba69" exitCode=0 Nov 03 15:07:40.887394 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:40.887315 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" event={"ID":"fabcd339c8e2b42bb3c3bff7d173edec","Type":"ContainerDied","Data":"cc03d66fb51322da4a1ff0f26ca5dc4080274054bd2a01a2289c9e445827ba69"} Nov 03 15:07:41.360998 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:41.360881 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" podStartSLOduration=6.360859359 podStartE2EDuration="6.360859359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:41.237836934 +0000 UTC m=+7.969125853" watchObservedRunningTime="2026-11-03 15:07:41.360859359 +0000 UTC m=+8.092148277" Nov 03 15:07:41.484252 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:41.484189 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q5rxt" podStartSLOduration=3.413539471 podStartE2EDuration="7.484170337s" totalImagesPullingTime="4.070630866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:41.361807944 +0000 UTC m=+8.093096865" watchObservedRunningTime="2026-11-03 15:07:41.484170337 +0000 UTC m=+8.215459256" Nov 03 15:07:41.898910 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:41.898871 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" event={"ID":"fabcd339c8e2b42bb3c3bff7d173edec","Type":"ContainerStarted","Data":"ed4e592e47b74b7af45d88c66247d0fa69fc6aa9c3290bcb7ad4ea8a68999e15"} Nov 03 15:07:42.661521 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:42.661019 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:42.661521 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:42.661117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:42.661521 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:42.661267 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:42.661521 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:42.661337 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret podName:f997ae6c-4cdd-4def-ac45-e81c30896420 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:50.661317348 +0000 UTC m=+17.392606261 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret") pod "global-pull-secret-syncer-hwchb" (UID: "f997ae6c-4cdd-4def-ac45-e81c30896420") : object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:42.661521 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:42.661406 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:42.661521 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:42.661443 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs podName:a6fc4400-11f3-42a8-afa3-68143e29eb10 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:50.661429438 +0000 UTC m=+17.392718335 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs") pod "network-metrics-daemon-fkx68" (UID: "a6fc4400-11f3-42a8-afa3-68143e29eb10") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:42.762295 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:42.762263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4st\" (UniqueName: \"kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st\") pod \"network-check-target-qwzv7\" (UID: \"9d58285c-7e8d-4833-87a2-66ba8f398c8d\") " pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:42.762584 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:42.762454 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 15:07:42.762584 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:42.762480 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 15:07:42.762584 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:42.762494 2575 projected.go:196] Error preparing data for projected volume kube-api-access-9m4st for pod openshift-network-diagnostics/network-check-target-qwzv7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:42.762584 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:42.762544 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st podName:9d58285c-7e8d-4833-87a2-66ba8f398c8d nodeName:}" failed. No retries permitted until 2026-11-03 15:07:50.762526552 +0000 UTC m=+17.493815451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m4st" (UniqueName: "kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st") pod "network-check-target-qwzv7" (UID: "9d58285c-7e8d-4833-87a2-66ba8f398c8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:42.813643 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:42.813430 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:42.813643 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:42.813457 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:42.813643 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:42.813582 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:07:42.813934 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:42.813662 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:07:42.813934 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:42.813748 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:42.813934 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:42.813840 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:07:42.903548 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:42.903484 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7fz2g" event={"ID":"ab771653-7ecd-4022-900f-c02f580bbb01","Type":"ContainerStarted","Data":"c58c9dc16483d5a83a02c68a45d3f7450fa66a69767e58f6c3930fe527e58753"} Nov 03 15:07:43.030768 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:43.030001 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" podStartSLOduration=8.029981504 podStartE2EDuration="8.029981504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:42.02609987 +0000 UTC m=+8.757388791" watchObservedRunningTime="2026-11-03 15:07:43.029981504 +0000 UTC m=+9.761270419" Nov 03 15:07:43.907840 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:43.907799 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" event={"ID":"b304a74d-882a-44e8-a696-4aff7665157c","Type":"ContainerStarted","Data":"7055dc258e948fbf2c846997a975d8e8db5917d87b7e654a2a7f32b7e593da8e"} Nov 03 15:07:44.167692 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:44.167619 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:07:44.169137 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:44.169109 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:07:44.293011 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:44.292943 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7fz2g" podStartSLOduration=4.487467808 podStartE2EDuration="10.292923731s" totalImagesPullingTime="5.805455923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:43.030751216 +0000 UTC m=+9.762040136" watchObservedRunningTime="2026-11-03 15:07:44.292923731 +0000 UTC m=+11.024212649" Nov 03 15:07:44.813310 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:44.813272 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:44.813503 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:44.813411 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:07:44.813567 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:44.813499 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:44.813615 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:44.813586 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:07:44.813669 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:44.813662 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:44.813801 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:44.813731 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:07:45.171250 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:45.171219 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 15:07:45.684474 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:45.684387 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:07:45.685411 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:45.685109 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:07:45.751368 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:45.751266 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T15:07:45.17124533Z","UUID":"9f25b5ab-81a6-4797-a944-7abb33db1ecc","Handler":null,"Name":"","Endpoint":""} Nov 03 15:07:45.753178 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:45.753156 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 15:07:45.753317 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:45.753187 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 15:07:45.919157 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:45.918863 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" event={"ID":"b304a74d-882a-44e8-a696-4aff7665157c","Type":"ContainerStarted","Data":"78175261e1955db832178ad87c582b1502ce051b807c0e0187f2d99ace3cd9dc"} Nov 03 15:07:45.921286 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:45.921256 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dtkj5" event={"ID":"c46a1f0f-4dc2-4b03-8269-d19a93977b56","Type":"ContainerStarted","Data":"1ecf1996aca35be11ed37bf6de2ff3337c9e779d36b125524ee72381ffaf15df"} Nov 03 15:07:45.924398 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:45.924370 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cmv4d" event={"ID":"588c75cb-ff10-45c1-be72-625f6d9b9d9e","Type":"ContainerStarted","Data":"bc5061cfeb2ef5d92fe0131306fce913ca31a58180c6ada120c7d642e5801b3c"} Nov 03 15:07:45.928148 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:45.928119 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkw2x" event={"ID":"5c8010d3-c54b-48b1-b792-fbd8933863a2","Type":"ContainerStarted","Data":"fb2e1970cfc9b97a5f9aafed1eef353206c4e8c56297add225fa581c60e141a2"} Nov 03 15:07:46.172274 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:46.172217 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cmv4d" podStartSLOduration=3.105297152 podStartE2EDuration="12.172199838s" totalImagesPullingTime="9.066902686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:46.172000968 +0000 UTC m=+12.903289887" watchObservedRunningTime="2026-11-03 15:07:46.172199838 +0000 UTC m=+12.903488757" Nov 03 15:07:46.172711 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:46.172535 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-dtkj5" podStartSLOduration=3.503015546 podStartE2EDuration="12.172528946s" totalImagesPullingTime="8.6695134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:46.048846608 +0000 UTC m=+12.780135527" watchObservedRunningTime="2026-11-03 15:07:46.172528946 +0000 UTC m=+12.903817871" Nov 03 15:07:46.813300 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:46.812967 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:46.813300 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:46.812993 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:46.813300 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:46.813110 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:07:46.813300 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:46.813176 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:46.813300 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:46.813263 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:07:46.813300 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:46.813171 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:07:46.932207 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:46.932166 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" event={"ID":"b304a74d-882a-44e8-a696-4aff7665157c","Type":"ContainerStarted","Data":"95f140f17aaa8fe2013e4f34711d4e7c7a491ab67b1457b55c1f2c3b73406e19"} Nov 03 15:07:46.934323 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:46.934292 2575 generic.go:356] "Generic (PLEG): container finished" podID="5c8010d3-c54b-48b1-b792-fbd8933863a2" containerID="fb2e1970cfc9b97a5f9aafed1eef353206c4e8c56297add225fa581c60e141a2" exitCode=0 Nov 03 15:07:46.934438 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:46.934374 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkw2x" event={"ID":"5c8010d3-c54b-48b1-b792-fbd8933863a2","Type":"ContainerDied","Data":"fb2e1970cfc9b97a5f9aafed1eef353206c4e8c56297add225fa581c60e141a2"} Nov 03 15:07:46.938967 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:46.938893 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rq6pc" event={"ID":"7d5de266-de42-430c-b364-d12951764b40","Type":"ContainerStarted","Data":"2a91e2880a2776529346d555cba20d209ca4e84c90b60cce10eb13f99bf8fe85"} Nov 03 15:07:47.058558 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:47.058448 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" podStartSLOduration=2.844680956 podStartE2EDuration="13.058429391s" totalImagesPullingTime="10.213748435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:47.057937492 +0000 UTC m=+13.789226412" watchObservedRunningTime="2026-11-03 15:07:47.058429391 +0000 UTC m=+13.789718308" Nov 03 15:07:47.309117 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:47.309012 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rq6pc" podStartSLOduration=3.912517995 podStartE2EDuration="13.308999333s" totalImagesPullingTime="9.396481338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:47.308481114 +0000 UTC m=+14.039770028" watchObservedRunningTime="2026-11-03 15:07:47.308999333 +0000 UTC m=+14.040288246" Nov 03 15:07:47.942351 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:47.942306 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" event={"ID":"a1129e2e-263b-48b8-a239-97d9164beca1","Type":"ContainerStarted","Data":"bf95a39b64b92cfe5788d2742939a66578c487217eb5694b956fa2501658e6d1"} Nov 03 15:07:48.058335 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:48.058280 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" podStartSLOduration=2.542195317 podStartE2EDuration="14.058262914s" totalImagesPullingTime="11.516067597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:48.057789422 +0000 UTC m=+14.789078337" watchObservedRunningTime="2026-11-03 15:07:48.058262914 +0000 UTC m=+14.789551832" Nov 03 15:07:48.812994 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:48.812961 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:48.813668 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:48.813048 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:48.813668 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:48.813166 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:07:48.813668 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:48.813201 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:48.813668 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:48.813267 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:07:48.813668 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:48.813347 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:07:48.947209 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:48.947176 2575 generic.go:356] "Generic (PLEG): container finished" podID="5c8010d3-c54b-48b1-b792-fbd8933863a2" containerID="7e0f09c1342efa2fff76f2d807615c8608e8cf1570a57ee7be7096bed86b932c" exitCode=0 Nov 03 15:07:48.947358 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:48.947252 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkw2x" event={"ID":"5c8010d3-c54b-48b1-b792-fbd8933863a2","Type":"ContainerDied","Data":"7e0f09c1342efa2fff76f2d807615c8608e8cf1570a57ee7be7096bed86b932c"} Nov 03 15:07:50.737965 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:50.737715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:50.738509 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:50.738011 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:50.738509 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:50.737865 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:50.738509 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:50.738101 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:50.738509 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:50.738114 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret podName:f997ae6c-4cdd-4def-ac45-e81c30896420 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:06.738100514 +0000 UTC m=+33.469389407 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret") pod "global-pull-secret-syncer-hwchb" (UID: "f997ae6c-4cdd-4def-ac45-e81c30896420") : object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:50.738509 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:50.738153 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs podName:a6fc4400-11f3-42a8-afa3-68143e29eb10 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:06.738138789 +0000 UTC m=+33.469427696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs") pod "network-metrics-daemon-fkx68" (UID: "a6fc4400-11f3-42a8-afa3-68143e29eb10") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:50.812854 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:50.812815 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:50.812854 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:50.812852 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:50.813099 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:50.812818 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:50.813099 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:50.812936 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:07:50.813099 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:50.813028 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:07:50.813099 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:50.813092 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:07:50.838958 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:50.838931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4st\" (UniqueName: \"kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st\") pod \"network-check-target-qwzv7\" (UID: \"9d58285c-7e8d-4833-87a2-66ba8f398c8d\") " pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:50.839105 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:50.839050 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 15:07:50.839105 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:50.839066 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 15:07:50.839105 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:50.839077 2575 projected.go:196] Error preparing data for projected volume kube-api-access-9m4st for pod openshift-network-diagnostics/network-check-target-qwzv7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:50.839256 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:50.839124 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st podName:9d58285c-7e8d-4833-87a2-66ba8f398c8d nodeName:}" failed. No retries permitted until 2026-11-03 15:08:06.839106796 +0000 UTC m=+33.570395691 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m4st" (UniqueName: "kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st") pod "network-check-target-qwzv7" (UID: "9d58285c-7e8d-4833-87a2-66ba8f398c8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:50.955452 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:50.955416 2575 generic.go:356] "Generic (PLEG): container finished" podID="5c8010d3-c54b-48b1-b792-fbd8933863a2" containerID="593a153b04748dbd119c651a47b349cd9f8f42fe9e1fc71eddc67d602150e9fa" exitCode=0 Nov 03 15:07:50.955630 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:50.955474 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkw2x" event={"ID":"5c8010d3-c54b-48b1-b792-fbd8933863a2","Type":"ContainerDied","Data":"593a153b04748dbd119c651a47b349cd9f8f42fe9e1fc71eddc67d602150e9fa"} Nov 03 15:07:51.961595 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:51.961307 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" event={"ID":"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa","Type":"ContainerStarted","Data":"05dc330507b54038b2dab2cca396511d2c2b50e0839adf0e34152a218ad567d4"} Nov 03 15:07:51.962067 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:51.961600 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" event={"ID":"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa","Type":"ContainerStarted","Data":"62d1661f4fc6f5c969dc368526e047810c2e653d5e025f9faa1dc832d043382d"} Nov 03 15:07:51.962067 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:51.961635 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" event={"ID":"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa","Type":"ContainerStarted","Data":"924fe12d90705a1080eba9e2b3e526685828ad9fcd4cd0f57d3f718125ffcc61"} Nov 03 15:07:51.962067 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:51.961673 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" event={"ID":"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa","Type":"ContainerStarted","Data":"f585938b31b020b85e00e3c2517bd55f4f71f8488eaba94801a9adcde3d6e80e"} Nov 03 15:07:51.962067 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:51.961724 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" event={"ID":"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa","Type":"ContainerStarted","Data":"a27d757274c258e527111d247f2d355625db3401f1db5dbcecd093e2ea5a99c3"} Nov 03 15:07:51.962067 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:51.961769 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" event={"ID":"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa","Type":"ContainerStarted","Data":"79e5b6e9d99895244ec6a8af283ff0983d3c82700917cea335436631f461063e"} Nov 03 15:07:52.812950 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:52.812914 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:52.813124 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:52.813026 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:52.813124 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:52.813054 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:52.813217 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:52.813170 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:07:52.813294 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:52.813275 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:07:52.813382 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:52.813351 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:07:53.968420 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:53.968384 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" event={"ID":"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa","Type":"ContainerStarted","Data":"b6faa24cc032e937da19f102090edd78742b6058ce400e2c392683aca6abaafe"} Nov 03 15:07:53.971518 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:53.971490 2575 generic.go:356] "Generic (PLEG): container finished" podID="5c8010d3-c54b-48b1-b792-fbd8933863a2" containerID="629dbbcdcb7f8c786213edbcf6459ecab06787f0d246fce236377a60b9ae8535" exitCode=0 Nov 03 15:07:53.971637 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:53.971545 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkw2x" event={"ID":"5c8010d3-c54b-48b1-b792-fbd8933863a2","Type":"ContainerDied","Data":"629dbbcdcb7f8c786213edbcf6459ecab06787f0d246fce236377a60b9ae8535"} Nov 03 15:07:54.812569 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:54.812537 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:54.812569 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:54.812565 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:54.812768 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:54.812649 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:54.812768 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:54.812656 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:07:54.814329 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:54.813055 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:07:54.814329 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:54.813219 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:07:54.976358 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:54.976330 2575 generic.go:356] "Generic (PLEG): container finished" podID="5c8010d3-c54b-48b1-b792-fbd8933863a2" containerID="0b173cf78d8485ad2c56989a60237d03f57631cd0426cd16e3b0defbeac1b7bd" exitCode=0 Nov 03 15:07:54.976939 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:54.976367 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkw2x" event={"ID":"5c8010d3-c54b-48b1-b792-fbd8933863a2","Type":"ContainerDied","Data":"0b173cf78d8485ad2c56989a60237d03f57631cd0426cd16e3b0defbeac1b7bd"} Nov 03 15:07:55.981511 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:55.981478 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkw2x" event={"ID":"5c8010d3-c54b-48b1-b792-fbd8933863a2","Type":"ContainerStarted","Data":"79350bcba77dcc461531cc6478198795a752ed8ee295dcf57a7aad686633447f"} Nov 03 15:07:56.211971 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:56.211859 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zkw2x" podStartSLOduration=9.43632865 podStartE2EDuration="22.211846175s" totalImagesPullingTime="12.775517525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:56.211518539 +0000 UTC m=+22.942807447" watchObservedRunningTime="2026-11-03 15:07:56.211846175 +0000 UTC m=+22.943135089" Nov 03 15:07:56.813603 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:56.813429 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:56.813879 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:56.813447 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:56.813879 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:56.813497 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:56.813879 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:56.813758 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:07:56.813879 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:56.813762 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:07:56.813879 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:56.813831 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:07:56.986614 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:56.986579 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" event={"ID":"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa","Type":"ContainerStarted","Data":"931a6627284bcaa24a3c0870cb5ffc6be593e74981de7348db7a2b6b50a04cd2"} Nov 03 15:07:57.269230 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:57.269174 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" podStartSLOduration=8.144619176 podStartE2EDuration="23.269160487s" totalImagesPullingTime="15.124541311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:57.268672693 +0000 UTC m=+23.999961606" watchObservedRunningTime="2026-11-03 15:07:57.269160487 +0000 UTC m=+24.000449401" Nov 03 15:07:57.988906 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:57.988868 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:57.989294 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:57.988953 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:57.989294 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:57.988998 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:58.005422 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:58.005392 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:58.005547 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:58.005496 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:07:58.812752 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:58.812675 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:58.812752 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:58.812721 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:58.812922 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:58.812799 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:58.812922 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:58.812805 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:07:58.812922 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:58.812869 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:07:58.813012 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:58.812944 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:07:58.830529 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:58.830502 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hwchb"] Nov 03 15:07:58.834425 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:58.834401 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fkx68"] Nov 03 15:07:58.835081 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:58.835065 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qwzv7"] Nov 03 15:07:58.991056 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:58.991033 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:07:58.991056 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:58.991056 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:07:58.991506 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:07:58.991033 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:07:58.991506 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:58.991132 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:07:58.991681 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:58.991661 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:07:58.991768 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:07:58.991751 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:08:00.812945 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:00.812911 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:08:00.813334 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:08:00.813014 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:08:00.813334 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:00.813030 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:08:00.813334 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:00.812926 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:08:00.813334 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:08:00.813118 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:08:00.813458 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:08:00.813398 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:08:02.812574 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:02.812539 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:08:02.813010 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:02.812646 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:08:02.813010 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:08:02.812659 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwchb" podUID="f997ae6c-4cdd-4def-ac45-e81c30896420" Nov 03 15:08:02.813010 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:02.812671 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:08:02.813010 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:08:02.812751 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwzv7" podUID="9d58285c-7e8d-4833-87a2-66ba8f398c8d" Nov 03 15:08:02.813010 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:08:02.812841 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkx68" podUID="a6fc4400-11f3-42a8-afa3-68143e29eb10" Nov 03 15:08:04.078141 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:04.078099 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeReady" Nov 03 15:08:04.078471 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:04.078253 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 15:08:04.812848 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:04.812639 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:08:04.813002 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:04.812662 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:08:04.813060 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:04.812718 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:08:04.866843 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:04.866820 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 15:08:04.866843 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:04.866821 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 15:08:04.917584 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:04.917555 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-975kv\"" Nov 03 15:08:04.917696 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:04.917612 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 15:08:04.917696 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:04.917647 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qbvnh\"" Nov 03 15:08:04.917805 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:04.917727 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z77jb\"" Nov 03 15:08:04.917837 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:04.917803 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 15:08:06.751828 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:06.751791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:08:06.752186 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:06.751838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:08:06.752186 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:08:06.751948 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 15:08:06.752186 ip-10-0-139-35 kubenswrapper[2575]: E1103 15:08:06.752015 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs podName:a6fc4400-11f3-42a8-afa3-68143e29eb10 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:38.751997089 +0000 UTC m=+65.483286000 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs") pod "network-metrics-daemon-fkx68" (UID: "a6fc4400-11f3-42a8-afa3-68143e29eb10") : secret "metrics-daemon-secret" not found Nov 03 15:08:06.755630 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:06.755611 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:08:06.853017 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:06.852987 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4st\" (UniqueName: \"kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st\") pod \"network-check-target-qwzv7\" (UID: \"9d58285c-7e8d-4833-87a2-66ba8f398c8d\") " pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:08:06.855350 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:06.855331 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m4st\" (UniqueName: \"kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st\") pod \"network-check-target-qwzv7\" (UID: \"9d58285c-7e8d-4833-87a2-66ba8f398c8d\") " pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:08:06.930404 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:06.930367 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:08:06.934243 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:06.934226 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:08:07.310500 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:07.310472 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hwchb"] Nov 03 15:08:07.365327 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:07.365304 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qwzv7"] Nov 03 15:08:07.368421 ip-10-0-139-35 kubenswrapper[2575]: W1103 15:08:07.368400 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d58285c_7e8d_4833_87a2_66ba8f398c8d.slice/crio-74ef84d4626757adaa106a8ee67a2938ebea341d775646165fec18809a44615d WatchSource:0}: Error finding container 74ef84d4626757adaa106a8ee67a2938ebea341d775646165fec18809a44615d: Status 404 returned error can't find the container with id 74ef84d4626757adaa106a8ee67a2938ebea341d775646165fec18809a44615d Nov 03 15:08:08.010666 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:08.010640 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hwchb" event={"ID":"f997ae6c-4cdd-4def-ac45-e81c30896420","Type":"ContainerStarted","Data":"875176edbb830c26ae6d7f9a23ce4cbd44e24f9993f462f221b618785154efb7"} Nov 03 15:08:08.011606 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:08.011588 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qwzv7" event={"ID":"9d58285c-7e8d-4833-87a2-66ba8f398c8d","Type":"ContainerStarted","Data":"74ef84d4626757adaa106a8ee67a2938ebea341d775646165fec18809a44615d"} Nov 03 15:08:09.109879 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:09.109848 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cmv4d_588c75cb-ff10-45c1-be72-625f6d9b9d9e/dns-node-resolver/0.log" Nov 03 15:08:09.733032 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:09.733006 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7fz2g_ab771653-7ecd-4022-900f-c02f580bbb01/node-ca/0.log" Nov 03 15:08:11.021150 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:11.021111 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qwzv7" event={"ID":"9d58285c-7e8d-4833-87a2-66ba8f398c8d","Type":"ContainerStarted","Data":"7c8f7a0987c762c3166ce7b38c952cfaa33aa5ab0e3ac8c7f82a169b81a1eca9"} Nov 03 15:08:11.021591 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:11.021448 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:08:11.138844 ip-10-0-139-35 kubenswrapper[2575]: I1103 15:08:11.138790 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qwzv7" podStartSLOduration=33.731705661 podStartE2EDuration="37.138772903s" totalImagesPullingTime="3.407067242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:11.138632014 +0000 UTC m=+37.869920930" watchObservedRunningTime="2026-11-03 15:08:11.138772903 +0000 UTC m=+37.870061829" Nov 03 15:08:12.508272 ip-10-0-139-35 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 15:08:12.523117 ip-10-0-139-35 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 15:08:12.523283 ip-10-0-139-35 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 15:08:12.523331 ip-10-0-139-35 systemd[1]: kubelet.service: Consumed 2.056s CPU time. Nov 03 15:08:12.539363 ip-10-0-139-35 systemd[1]: Starting Kubernetes Kubelet... Nov 03 15:08:12.597961 ip-10-0-139-35 kubenswrapper[4290]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 15:08:12.597961 ip-10-0-139-35 kubenswrapper[4290]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 15:08:12.597961 ip-10-0-139-35 kubenswrapper[4290]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 15:08:12.597961 ip-10-0-139-35 kubenswrapper[4290]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 15:08:12.600034 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.599958 4290 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:12.600034 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600034 4290 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:12.600034 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600038 4290 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600041 4290 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600044 4290 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600047 4290 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600050 4290 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600052 4290 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600055 4290 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600057 4290 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600060 4290 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600063 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600065 4290 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600070 4290 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600075 4290 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600078 4290 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600081 4290 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600084 4290 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600086 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600089 4290 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600091 4290 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:12.600127 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600094 4290 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600097 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600103 4290 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600106 4290 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600109 4290 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600112 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600114 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600117 4290 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600119 4290 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600122 4290 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600124 4290 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600127 4290 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600130 4290 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600133 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600136 4290 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600138 4290 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600141 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600144 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600146 4290 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:12.600601 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600149 4290 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600151 4290 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600154 4290 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600157 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600160 4290 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600162 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600164 4290 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600167 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600170 4290 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600172 4290 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600174 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600177 4290 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600179 4290 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600182 4290 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600185 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600187 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600190 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600193 4290 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600196 4290 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600198 4290 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:12.601094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600200 4290 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600203 4290 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600206 4290 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600208 4290 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600211 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600213 4290 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600217 4290 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600220 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600223 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600225 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600227 4290 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600230 4290 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600233 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600235 4290 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600238 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600240 4290 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600248 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600251 4290 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600254 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:12.601575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600256 4290 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600260 4290 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600263 4290 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600266 4290 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600268 4290 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600271 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600273 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600276 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600279 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600281 4290 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600284 4290 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600287 4290 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600289 4290 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600292 4290 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600295 4290 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600297 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600300 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600302 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600305 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600307 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:12.602108 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600309 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600312 4290 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600315 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600317 4290 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600321 4290 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600323 4290 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.600325 4290 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601057 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601063 4290 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601067 4290 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601077 4290 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601081 4290 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601083 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601086 4290 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601089 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601092 4290 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601095 4290 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601097 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601100 4290 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:12.602602 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601103 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601106 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601109 4290 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601111 4290 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601115 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601118 4290 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601120 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601123 4290 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601125 4290 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601128 4290 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601131 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601133 4290 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601136 4290 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601138 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601141 4290 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601143 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601146 4290 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601148 4290 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601151 4290 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601153 4290 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:12.603086 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601155 4290 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601158 4290 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601160 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601163 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601166 4290 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601169 4290 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601172 4290 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601174 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601176 4290 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601179 4290 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601182 4290 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601184 4290 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601188 4290 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601191 4290 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601193 4290 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601196 4290 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601198 4290 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601201 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601204 4290 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601206 4290 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:12.603566 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601209 4290 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601211 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601214 4290 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601216 4290 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601219 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601221 4290 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601223 4290 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601226 4290 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601228 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601231 4290 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601233 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601235 4290 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601239 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601241 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601244 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601247 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601249 4290 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601252 4290 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601255 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601257 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:12.604085 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601260 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601262 4290 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601265 4290 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601268 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601270 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601273 4290 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601275 4290 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601278 4290 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601280 4290 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601283 4290 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601285 4290 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601288 4290 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601290 4290 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601292 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601295 4290 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601297 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601300 4290 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601302 4290 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601305 4290 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:12.604575 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601308 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601310 4290 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601313 4290 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601315 4290 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601318 4290 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601320 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601322 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601325 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601327 4290 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601330 4290 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601334 4290 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601339 4290 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601343 4290 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601346 4290 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601349 4290 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601825 4290 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601832 4290 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601835 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601838 4290 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:12.605068 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601841 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601843 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601846 4290 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601849 4290 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601851 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601854 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601856 4290 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601859 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601862 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601864 4290 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601867 4290 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601870 4290 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601872 4290 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601876 4290 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601880 4290 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601883 4290 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601886 4290 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601889 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601891 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:12.605579 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601894 4290 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601896 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601899 4290 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601901 4290 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601904 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601908 4290 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601910 4290 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601913 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601916 4290 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601923 4290 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601925 4290 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601928 4290 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601931 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601933 4290 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601936 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601938 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601941 4290 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601943 4290 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601946 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601948 4290 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:12.606084 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601950 4290 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601953 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601956 4290 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601958 4290 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601961 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601963 4290 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601966 4290 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601969 4290 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601971 4290 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601974 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601976 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601979 4290 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601982 4290 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601984 4290 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601987 4290 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601990 4290 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601992 4290 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601994 4290 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.601997 4290 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602000 4290 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:12.606581 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602002 4290 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602006 4290 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602009 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602011 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602014 4290 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602016 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602019 4290 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602021 4290 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602024 4290 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602026 4290 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602029 4290 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602031 4290 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602034 4290 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602036 4290 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602039 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602041 4290 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602045 4290 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602048 4290 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602051 4290 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:12.607080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602053 4290 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602055 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602058 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602060 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602063 4290 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602065 4290 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602068 4290 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602070 4290 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602073 4290 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602076 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602078 4290 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602081 4290 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602084 4290 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602086 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602090 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602092 4290 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602095 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602097 4290 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602100 4290 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602102 4290 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:12.607589 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602105 4290 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602107 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602110 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602112 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602205 4290 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602213 4290 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602219 4290 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602223 4290 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602227 4290 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602231 4290 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602236 4290 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602240 4290 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602243 4290 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602245 4290 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602249 4290 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602252 4290 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602255 4290 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602258 4290 flags.go:64] FLAG: --cgroup-root="" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602261 4290 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602264 4290 flags.go:64] FLAG: --client-ca-file="" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602266 4290 flags.go:64] FLAG: --cloud-provider="external" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602269 4290 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602273 4290 flags.go:64] FLAG: --cluster-domain="" Nov 03 15:08:12.608092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602276 4290 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602279 4290 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602283 4290 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602287 4290 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602291 4290 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602294 4290 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602297 4290 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602300 4290 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602303 4290 flags.go:64] FLAG: --contention-profiling="false" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602305 4290 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602308 4290 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602311 4290 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602314 4290 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602318 4290 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602321 4290 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602324 4290 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602326 4290 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602329 4290 flags.go:64] FLAG: --enable-server="true" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602334 4290 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602338 4290 flags.go:64] FLAG: --event-burst="100" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602341 4290 flags.go:64] FLAG: --event-qps="50" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602344 4290 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602347 4290 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602349 4290 flags.go:64] FLAG: --eviction-hard="" Nov 03 15:08:12.608842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602353 4290 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602356 4290 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602359 4290 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602362 4290 flags.go:64] FLAG: --eviction-soft="" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602365 4290 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602368 4290 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602371 4290 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602374 4290 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602377 4290 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602380 4290 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602383 4290 flags.go:64] FLAG: --feature-gates="" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602387 4290 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602390 4290 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602393 4290 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602396 4290 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602399 4290 flags.go:64] FLAG: --healthz-port="10248" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602402 4290 flags.go:64] FLAG: --help="false" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602405 4290 flags.go:64] FLAG: --hostname-override="ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602408 4290 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602411 4290 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602414 4290 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602417 4290 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602420 4290 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 15:08:12.609747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602423 4290 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602425 4290 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602428 4290 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602433 4290 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602435 4290 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602438 4290 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602441 4290 flags.go:64] FLAG: --kube-reserved="" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602444 4290 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602447 4290 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602450 4290 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602452 4290 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602455 4290 flags.go:64] FLAG: --lock-file="" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602458 4290 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602460 4290 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602463 4290 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602469 4290 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602472 4290 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602475 4290 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602477 4290 flags.go:64] FLAG: --logging-format="text" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602480 4290 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602483 4290 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602491 4290 flags.go:64] FLAG: --manifest-url="" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602494 4290 flags.go:64] FLAG: --manifest-url-header="" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602498 4290 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602501 4290 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 15:08:12.610307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602504 4290 flags.go:64] FLAG: --max-pods="110" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602507 4290 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602510 4290 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602513 4290 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602515 4290 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602518 4290 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602521 4290 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602524 4290 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602530 4290 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602533 4290 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602537 4290 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602540 4290 flags.go:64] FLAG: --pod-cidr="" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602543 4290 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602546 4290 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602549 4290 flags.go:64] FLAG: --pods-per-core="0" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602552 4290 flags.go:64] FLAG: --port="10250" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602555 4290 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602557 4290 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0321c9227970242b8" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602561 4290 flags.go:64] FLAG: --qos-reserved="" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602563 4290 flags.go:64] FLAG: --read-only-port="10255" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602566 4290 flags.go:64] FLAG: --register-node="true" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602569 4290 flags.go:64] FLAG: --register-with-taints="" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602573 4290 flags.go:64] FLAG: --registry-burst="10" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602575 4290 flags.go:64] FLAG: --registry-qps="5" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602578 4290 flags.go:64] FLAG: --reserved-cpus="" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602581 4290 flags.go:64] FLAG: --reserved-memory="" Nov 03 15:08:12.610916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602585 4290 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602588 4290 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602590 4290 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602594 4290 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602596 4290 flags.go:64] FLAG: --runonce="false" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602599 4290 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602602 4290 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602605 4290 flags.go:64] FLAG: --seccomp-default="false" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602608 4290 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602611 4290 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602614 4290 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602617 4290 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602620 4290 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602622 4290 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602625 4290 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602628 4290 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602632 4290 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602635 4290 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602638 4290 flags.go:64] FLAG: --system-cgroups="" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602641 4290 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602646 4290 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602648 4290 flags.go:64] FLAG: --tls-cert-file="" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602652 4290 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602656 4290 flags.go:64] FLAG: --tls-min-version="" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602659 4290 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 15:08:12.611575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602662 4290 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602665 4290 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602668 4290 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602671 4290 flags.go:64] FLAG: --v="2" Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602675 4290 flags.go:64] FLAG: --version="false" Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602679 4290 flags.go:64] FLAG: --vmodule="" Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602682 4290 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.602685 4290 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602858 4290 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602863 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602867 4290 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602871 4290 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602874 4290 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602877 4290 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602880 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602882 4290 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602885 4290 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602888 4290 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602890 4290 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602893 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602895 4290 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602898 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:12.612193 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602900 4290 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602907 4290 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602910 4290 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602912 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602915 4290 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602918 4290 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602920 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602925 4290 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602930 4290 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602933 4290 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602935 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602938 4290 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602940 4290 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602943 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602945 4290 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602948 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602950 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602953 4290 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602956 4290 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602958 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:12.612971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602961 4290 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602963 4290 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602967 4290 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602970 4290 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602972 4290 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602975 4290 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602977 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602980 4290 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602982 4290 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602984 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602987 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602989 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602992 4290 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602996 4290 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.602998 4290 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603001 4290 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603003 4290 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603006 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603009 4290 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603012 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:12.613620 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603015 4290 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603017 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603020 4290 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603022 4290 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603025 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603027 4290 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603030 4290 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603033 4290 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603037 4290 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603040 4290 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603043 4290 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603045 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603048 4290 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603050 4290 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603053 4290 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603056 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603059 4290 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603062 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603064 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:12.614202 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603067 4290 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603069 4290 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603071 4290 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603074 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603076 4290 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603079 4290 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603083 4290 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603085 4290 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603088 4290 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603091 4290 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603093 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603096 4290 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603099 4290 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603102 4290 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603105 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603107 4290 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603110 4290 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603113 4290 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603115 4290 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:12.614690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603117 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:12.615176 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603120 4290 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:12.615176 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603122 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:12.615176 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603125 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:12.615176 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603127 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:12.615176 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603130 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:12.615176 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603132 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:12.615176 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603135 4290 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:12.615176 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603138 4290 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:12.615176 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603140 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:12.615176 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603143 4290 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:12.615176 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603145 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:12.615176 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603148 4290 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:12.615176 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603151 4290 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:12.615176 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.603175 4290 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 15:08:12.615176 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603209 4290 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603213 4290 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603216 4290 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603219 4290 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603223 4290 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603226 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603228 4290 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603231 4290 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603233 4290 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603236 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603238 4290 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603241 4290 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603244 4290 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603247 4290 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603249 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603252 4290 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603254 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603256 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603259 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:12.615608 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603261 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603264 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603266 4290 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603269 4290 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603271 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603274 4290 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603276 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603279 4290 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603281 4290 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603287 4290 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603290 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603293 4290 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603295 4290 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603298 4290 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603300 4290 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603303 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603305 4290 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603309 4290 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603311 4290 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603314 4290 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:12.616422 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603316 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603318 4290 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603321 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603323 4290 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603327 4290 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603329 4290 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603332 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603334 4290 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603337 4290 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603339 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603343 4290 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603347 4290 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603349 4290 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603352 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603355 4290 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603357 4290 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603360 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603362 4290 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603365 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:12.617283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603367 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603370 4290 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603373 4290 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603376 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603378 4290 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603381 4290 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603383 4290 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603386 4290 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603388 4290 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603391 4290 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603394 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603397 4290 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603399 4290 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603401 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603404 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603406 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603409 4290 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603412 4290 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603414 4290 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603417 4290 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:12.618107 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603420 4290 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603423 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603425 4290 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603428 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603430 4290 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603432 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603435 4290 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603437 4290 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603440 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603442 4290 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603445 4290 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603448 4290 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603452 4290 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603456 4290 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603458 4290 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603461 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603464 4290 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603466 4290 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603469 4290 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:12.619001 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603471 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603474 4290 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603476 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603480 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603483 4290 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603485 4290 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603488 4290 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603491 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.603493 4290 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.607462 4290 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.607475 4290 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.607492 4290 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.607496 4290 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607547 4290 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607552 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607555 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607558 4290 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607561 4290 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607563 4290 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:12.619826 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607566 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607568 4290 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607571 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607574 4290 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607576 4290 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607578 4290 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607581 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607584 4290 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607586 4290 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607589 4290 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607592 4290 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607595 4290 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607598 4290 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607600 4290 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607603 4290 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607605 4290 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607608 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607611 4290 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:12.620719 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607613 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607617 4290 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607621 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607625 4290 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607628 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607631 4290 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607634 4290 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607637 4290 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607640 4290 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607642 4290 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607644 4290 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607647 4290 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607649 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607652 4290 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607654 4290 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607657 4290 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607660 4290 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607662 4290 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607664 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607667 4290 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:12.621512 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607669 4290 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607672 4290 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607674 4290 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607676 4290 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607679 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607682 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607685 4290 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607687 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607690 4290 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607692 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607712 4290 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607716 4290 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607719 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607721 4290 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607724 4290 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607727 4290 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607729 4290 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607732 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607735 4290 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607739 4290 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:12.622432 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607742 4290 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607745 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607748 4290 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607750 4290 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607753 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607756 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607758 4290 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607760 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607763 4290 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607765 4290 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607768 4290 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607770 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607772 4290 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607775 4290 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607777 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607779 4290 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607782 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607784 4290 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607787 4290 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607790 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:12.623520 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607793 4290 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607796 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607798 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607800 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607803 4290 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607805 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607808 4290 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607810 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607813 4290 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607815 4290 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607818 4290 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607820 4290 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607823 4290 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607825 4290 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607827 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607830 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607832 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607835 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607837 4290 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607839 4290 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:12.624451 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607842 4290 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.607844 4290 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.607871 4290 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608000 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608005 4290 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608008 4290 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608011 4290 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608014 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608016 4290 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608019 4290 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608022 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608025 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608028 4290 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608030 4290 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608033 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608035 4290 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:12.625485 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608039 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608042 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608044 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608047 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608049 4290 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608052 4290 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608054 4290 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608057 4290 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608059 4290 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608062 4290 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608064 4290 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608067 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608070 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608072 4290 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608074 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608077 4290 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608079 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608082 4290 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608085 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608087 4290 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:12.626382 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608089 4290 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608092 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608094 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608097 4290 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608100 4290 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608102 4290 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608105 4290 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608108 4290 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608111 4290 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608114 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608116 4290 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608118 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608122 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608125 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608127 4290 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608130 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608132 4290 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608135 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608137 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:12.627251 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608140 4290 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608142 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608146 4290 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608149 4290 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608152 4290 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608155 4290 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608157 4290 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608160 4290 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608162 4290 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608165 4290 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608167 4290 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608169 4290 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608172 4290 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608175 4290 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608177 4290 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608179 4290 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608182 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608184 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608187 4290 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:12.627985 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608189 4290 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608191 4290 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608194 4290 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608196 4290 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608199 4290 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608202 4290 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608205 4290 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608208 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608211 4290 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608214 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608216 4290 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608219 4290 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608221 4290 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608224 4290 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608226 4290 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608229 4290 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608231 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608234 4290 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608236 4290 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:12.628642 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608240 4290 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608244 4290 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608247 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608250 4290 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608252 4290 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608254 4290 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608257 4290 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608259 4290 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608262 4290 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608265 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608267 4290 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608269 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608272 4290 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608275 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608277 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608280 4290 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:12.629138 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.608302 4290 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608363 4290 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608478 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608484 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608493 4290 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608499 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608503 4290 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608509 4290 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608514 4290 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608518 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608522 4290 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608526 4290 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608531 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608535 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608546 4290 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608550 4290 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608554 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608558 4290 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608563 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:12.629818 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608567 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608571 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608576 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608580 4290 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608584 4290 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608588 4290 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608592 4290 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608597 4290 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608608 4290 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608613 4290 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608617 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608621 4290 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608626 4290 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608630 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608634 4290 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608639 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608644 4290 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608650 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608654 4290 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608658 4290 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:12.630389 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608667 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608671 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608676 4290 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608680 4290 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608685 4290 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608689 4290 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608693 4290 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608718 4290 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608723 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608727 4290 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608731 4290 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608735 4290 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608740 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608749 4290 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608753 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608758 4290 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608762 4290 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608766 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608770 4290 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:12.630875 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608775 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608782 4290 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608789 4290 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608794 4290 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608800 4290 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608805 4290 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608809 4290 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608818 4290 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608823 4290 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608828 4290 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608835 4290 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608840 4290 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608844 4290 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608883 4290 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608889 4290 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608895 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608900 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608904 4290 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608909 4290 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:12.631323 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608918 4290 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608922 4290 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608927 4290 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608931 4290 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608935 4290 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608939 4290 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608943 4290 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608947 4290 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608954 4290 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608959 4290 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608963 4290 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608967 4290 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608976 4290 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608980 4290 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608985 4290 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608989 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608993 4290 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.608998 4290 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.609005 4290 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:12.631790 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.609010 4290 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.609016 4290 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.609021 4290 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.609025 4290 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.609031 4290 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.609042 4290 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.609046 4290 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.609051 4290 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.609055 4290 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.609060 4290 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:12.609066 4290 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.609172 4290 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.610537 4290 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.610594 4290 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.610802 4290 server.go:1014] "Starting client certificate rotation" Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.610935 4290 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 15:08:12.632257 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.610985 4290 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 15:02:34 +0000 UTC" deadline="2028-04-28 12:22:26.092552834 +0000 UTC" Nov 03 15:08:12.632642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.611016 4290 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13005h14m13.481540706s" Nov 03 15:08:12.632642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.611234 4290 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 15:08:12.632642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.612280 4290 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 15:08:12.632642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.613818 4290 log.go:25] "Validated CRI v1 runtime API" Nov 03 15:08:12.632642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.615179 4290 log.go:25] "Validated CRI v1 image API" Nov 03 15:08:12.632642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.615906 4290 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 15:08:12.632642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.617818 4290 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8da7b8f2-04e3-469c-bf59-3c0b6013c092:/dev/nvme0n1p4 bf265bef-b8b4-48d9-80f0-c6cb0507c88b:/dev/nvme0n1p3] Nov 03 15:08:12.632862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.617845 4290 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2203c5ec29ce08b35beabfed77b82c8f6a43e3ea6ef95f99a664fe286b6e7a7f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2203c5ec29ce08b35beabfed77b82c8f6a43e3ea6ef95f99a664fe286b6e7a7f/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e6b796d62ff9620023eec074e3548c07fc17e74f482e89cfef2cd50af363e29/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e6b796d62ff9620023eec074e3548c07fc17e74f482e89cfef2cd50af363e29/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57ee6785f2343b55754b9bb01099fcfcc9ce3178cde783c74cbc79c9349d31e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57ee6785f2343b55754b9bb01099fcfcc9ce3178cde783c74cbc79c9349d31e2/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/63a79073cf5f69c69bc4d1d8f29085e892cbe545d9f536ec9217b224fe6aafcb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/63a79073cf5f69c69bc4d1d8f29085e892cbe545d9f536ec9217b224fe6aafcb/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f735fd43670028f25e285734a6ac5bd76a2753e3c11ab0687aa1864def51645/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f735fd43670028f25e285734a6ac5bd76a2753e3c11ab0687aa1864def51645/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74ef84d4626757adaa106a8ee67a2938ebea341d775646165fec18809a44615d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74ef84d4626757adaa106a8ee67a2938ebea341d775646165fec18809a44615d/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/875176edbb830c26ae6d7f9a23ce4cbd44e24f9993f462f221b618785154efb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/875176edbb830c26ae6d7f9a23ce4cbd44e24f9993f462f221b618785154efb7/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba30a154663ef205fe454179476cf3a18853919cd8be2da60aa2efe644c29903/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba30a154663ef205fe454179476cf3a18853919cd8be2da60aa2efe644c29903/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4baa7e38dc58d63830d401dc214d8a46d3d9ace1005eb075ba9d4483631926b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4baa7e38dc58d63830d401dc214d8a46d3d9ace1005eb075ba9d4483631926b/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e84d1b045ea2891383e88467e2f57f1b375d351d948f188df591e46e31c27799/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e84d1b045ea2891383e88467e2f57f1b375d351d948f188df591e46e31c27799/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb974c8097603b0176a5da2b7104a65434f16cd1daf03f37c58fd18b7644b6f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb974c8097603b0176a5da2b7104a65434f16cd1daf03f37c58fd18b7644b6f0/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc55f4153c09d413dbf721ad9a66592d3147096497da3502bf7728d943bc3415/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc55f4153c09d413dbf721ad9a66592d3147096497da3502bf7728d943bc3415/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/588c75cb-ff10-45c1-be72-625f6d9b9d9e/volumes/kubernetes.io~projected/kube-api-access-dq5ct:{mountpoint:/var/lib/kubelet/pods/588c75cb-ff10-45c1-be72-625f6d9b9d9e/volumes/kubernetes.io~projected/kube-api-access-dq5ct major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c8010d3-c54b-48b1-b792-fbd8933863a2/volumes/kubernetes.io~projected/kube-api-access-5xpq6:{mountpoint:/var/lib/kubelet/pods/5c8010d3-c54b-48b1-b792-fbd8933863a2/volumes/kubernetes.io~projected/kube-api-access-5xpq6 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d5de266-de42-430c-b364-d12951764b40/volumes/kubernetes.io~projected/kube-api-access-xfpfv:{mountpoint:/var/lib/kubelet/pods/7d5de266-de42-430c-b364-d12951764b40/volumes/kubernetes.io~projected/kube-api-access-xfpfv major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d58285c-7e8d-4833-87a2-66ba8f398c8d/volumes/kubernetes.io~projected/kube-api-access-9m4st:{mountpoint:/var/lib/kubelet/pods/9d58285c-7e8d-4833-87a2-66ba8f398c8d/volumes/kubernetes.io~projected/kube-api-access-9m4st major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1129e2e-263b-48b8-a239-97d9164beca1/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a1129e2e-263b-48b8-a239-97d9164beca1/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1129e2e-263b-48b8-a239-97d9164beca1/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a1129e2e-263b-48b8-a239-97d9164beca1/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1129e2e-263b-48b8-a239-97d9164beca1/volumes/kubernetes.io~projected/kube-api-access-g7pzg:{mountpoint:/var/lib/kubelet/pods/a1129e2e-263b-48b8-a239-97d9164beca1/volumes/kubernetes.io~projected/kube-api-access-g7pzg major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6fc4400-11f3-42a8-afa3-68143e29eb10/volumes/kubernetes.io~projected/kube-api-access-8szqq:{mountpoint:/var/lib/kubelet/pods/a6fc4400-11f3-42a8-afa3-68143e29eb10/volumes/kubernetes.io~projected/kube-api-access-8szqq major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab771653-7ecd-4022-900f-c02f580bbb01/volumes/kubernetes.io~projected/kube-api-access-bwhk7:{mountpoint:/var/lib/kubelet/pods/ab771653-7ecd-4022-900f-c02f580bbb01/volumes/kubernetes.io~projected/kube-api-access-bwhk7 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b304a74d-882a-44e8-a696-4aff7665157c/volumes/kubernetes.io~projected/kube-api-access-vdj7s:{mountpoint:/var/lib/kubelet/pods/b304a74d-882a-44e8-a696-4aff7665157c/volumes/kubernetes.io~projected/kube-api-access-vdj7s major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c46a1f0f-4dc2-4b03-8269-d19a93977b56/volumes/kubernetes.io~projected/kube-api-access-tgnz5:{mountpoint:/var/lib/kubelet/pods/c46a1f0f-4dc2-4b03-8269-d19a93977b56/volumes/kubernetes.io~projected/kube-api-access-tgnz5 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4f18ec9-6d01-4a3d-9845-58b607a6225f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c4f18ec9-6d01-4a3d-9845-58b607a6225f/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa/volumes/kubernetes.io~projected/kube-api-access-p6zql:{mountpoint:/var/lib/kubelet/pods/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa/volumes/kubernetes.io~projected/kube-api-access-p6zql major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f997ae6c-4cdd-4def-ac45-e81c30896420/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f997ae6c-4cdd-4def-ac45-e81c30896420/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1d49de89e0814fbf8f2325ab9ab61e613e4042138210521c72c52f05c28b5095/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/303dbba95d6ca6a81c0e9cab7ed8c11734a0197943cb42822ed17c0ceee45bc9/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a2518531db9cced39c6395cb7592cbece3e98a8f8cf5c411704e619c85ef0915/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f3327578dd3ae98343a4164521fdbe0e9403ce22e4182b986b708f41ef6647d7/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/b3a82e16b7bc2b8408ee9027b77608977b9763a0ebbd47c72e3fd1c941cf4a20/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/4acb6c4b6bc2c5bd59be9390ad4696337a3602983aad35a041651682faedc2ee/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/9f599501e996b1afe36046aa74c477c364067cf31419fb4dbb79e27c3d6cd9eb/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/34c7878ee975301c21cf5b4cf3f04300ef8a52d7237825df459138505b104435/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/642911bbde7358c32ffaef8ad7c3966f0932334925195abab170405660b2b3b3/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/8feb3b9273f59e039ec71c167d800f415d7cc93ced645a0a2c2084d0e2936b7e/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/de32abe256bc663588fec6b1f4afaec26c09670c414612c7fd70b74e19af97cb/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/537cc3b63166c935886e9a5741d50f5358da005ab84bf27fca86d0862e34245f/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/05353205ce7386fa566027f7d1f2cddf5823645dcd039e5f021161d18e5e057c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/68d054eafdbe2174e3ce9a6b31ccffd47a57962f0220db330520cc52619d113f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/26f95c4c962a8488375f2e853eb3c0128afbc2f63ec823f00f7b169c1efafb2e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/76115eadb6d335c674c215975e8a69773ab77fc8dc13b780c38dee828bf3ee5e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9afe1747834d94a0b5e9898759b1bfacc1492f033b1f8e961f15ee2c35a6e6cb/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/634ec9232a66346cc994c609748b0ce5f4aa20fb6f570388f3b8b0fc4f147ae5/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/1add4f657d4bffa87d987c7c32144bc083f34c237b49adeebe7f04b292496277/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/48a30a34346bf55fbabc0c9d305518e0ad20ba44223452f4273cd8ebadd8067e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/155a08424a946ddcbe38652ed09dfc66782e358b943840f6c781c946944c19a3/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/f6598ef98b37cc9a5a742ec4e1f4ce26eb60e0a5156389a17ebea8a18d271ab2/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/43868db9dbd53ee5665821e57bce1b61ecfa932683af6fa5ca7f220d6a6b5220/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/e9e4cfa90f0d519254efd77b516686df655e24e241b0866ccd0c00b7e8fbe2f3/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/b315b55ce85f7200eb9b330aee79a81f29b259f53db7e4094dab592063b4fa31/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/edc18c6d8191184b65252012c57c5b3dbd5719c4740fc752d9d641247d59ff30/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/aced289c3522250ce8a6ba215413c4f0a1da4e5e3768fa4a1f84850a85dda6fb/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/96cc633f46d0c47c539685e8de04d6b9c6090c4fc0f6b84361a2e21aff2f2154/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/80fdb784cc1eb6464ce285c8803ec4f9f25233219e99a83384888c10398cc15d/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/9e3e67c5274c45eddaae49867ddf03f0592f525c171f94ae53059f70184d1a27/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/107c1fb095ce95dce54143aa8581a4a10e3456dd9833efc37c9588459f089737/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/a4b2fd19e9d13058a286ed819d9d09bd1439baf07e77f03e6283994e14c75267/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/7ad9625ed925d704bd910cc2e9b10e2fbb46d175edc2eb07ba615e1ee7003dc7/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a9a90f1bf61d5fd40de362751a5dce631a15af1ef532776f24df051c72e0a819/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/3bf85a0eb485ecd6df66dcfe7359624005b75aa6ebbfac3f699d67387c25f5a3/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 15:08:12.632862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.625965 4290 manager.go:217] Machine: {Timestamp:2026-11-03 15:08:12.625184019 +0000 UTC m=+0.056141831 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3128320 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27b9e33916635998834a3aa03e3cc5 SystemUUID:ec27b9e3-3916-6359-9883-4a3aa03e3cc5 BootID:dadb4ceb-9c1b-4553-8b25-ad12193082c8 Filesystems:[{Device:/run/containers/storage/overlay-containers/74ef84d4626757adaa106a8ee67a2938ebea341d775646165fec18809a44615d/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a1129e2e-263b-48b8-a239-97d9164beca1/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ab771653-7ecd-4022-900f-c02f580bbb01/volumes/kubernetes.io~projected/kube-api-access-bwhk7 DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1129e2e-263b-48b8-a239-97d9164beca1/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5c8010d3-c54b-48b1-b792-fbd8933863a2/volumes/kubernetes.io~projected/kube-api-access-5xpq6 DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e84d1b045ea2891383e88467e2f57f1b375d351d948f188df591e46e31c27799/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f997ae6c-4cdd-4def-ac45-e81c30896420/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a1129e2e-263b-48b8-a239-97d9164beca1/volumes/kubernetes.io~projected/kube-api-access-g7pzg DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb974c8097603b0176a5da2b7104a65434f16cd1daf03f37c58fd18b7644b6f0/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6fc4400-11f3-42a8-afa3-68143e29eb10/volumes/kubernetes.io~projected/kube-api-access-8szqq DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e6b796d62ff9620023eec074e3548c07fc17e74f482e89cfef2cd50af363e29/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2203c5ec29ce08b35beabfed77b82c8f6a43e3ea6ef95f99a664fe286b6e7a7f/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa/volumes/kubernetes.io~projected/kube-api-access-p6zql DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d58285c-7e8d-4833-87a2-66ba8f398c8d/volumes/kubernetes.io~projected/kube-api-access-9m4st DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7d5de266-de42-430c-b364-d12951764b40/volumes/kubernetes.io~projected/kube-api-access-xfpfv DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba30a154663ef205fe454179476cf3a18853919cd8be2da60aa2efe644c29903/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57ee6785f2343b55754b9bb01099fcfcc9ce3178cde783c74cbc79c9349d31e2/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b304a74d-882a-44e8-a696-4aff7665157c/volumes/kubernetes.io~projected/kube-api-access-vdj7s DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/588c75cb-ff10-45c1-be72-625f6d9b9d9e/volumes/kubernetes.io~projected/kube-api-access-dq5ct DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4baa7e38dc58d63830d401dc214d8a46d3d9ace1005eb075ba9d4483631926b/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc55f4153c09d413dbf721ad9a66592d3147096497da3502bf7728d943bc3415/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c46a1f0f-4dc2-4b03-8269-d19a93977b56/volumes/kubernetes.io~projected/kube-api-access-tgnz5 DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/875176edbb830c26ae6d7f9a23ce4cbd44e24f9993f462f221b618785154efb7/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c4f18ec9-6d01-4a3d-9845-58b607a6225f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/63a79073cf5f69c69bc4d1d8f29085e892cbe545d9f536ec9217b224fe6aafcb/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f735fd43670028f25e285734a6ac5bd76a2753e3c11ab0687aa1864def51645/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:74ef84d4626757a MacAddress:1a:5e:fd:3b:45:07 Speed:10000 Mtu:8901} {Name:875176edbb830c2 MacAddress:72:fd:ea:69:a3:ab Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:6d:0a:70:06:2b Speed:0 Mtu:9001} {Name:br-int MacAddress:de:a1:f7:68:b8:d8 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:6d:0a:70:06:2b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:92:25:6d:c6:e4:b9 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:02:3e:a5:45:bb:ed Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 15:08:12.632862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.626344 4290 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 15:08:12.632862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.626397 4290 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 15:08:12.632862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.626731 4290 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 15:08:12.632862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.626759 4290 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-35.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 15:08:12.632862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.626938 4290 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 15:08:12.632862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.626951 4290 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 15:08:12.632862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.626965 4290 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 15:08:12.632862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.626992 4290 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 15:08:12.632862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.627096 4290 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 15:08:12.632862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.627310 4290 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 15:08:12.632862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.627451 4290 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 15:08:12.632862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.627493 4290 kubelet.go:493] "Attempting to sync node with API server" Nov 03 15:08:12.632862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.627508 4290 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.627521 4290 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.627534 4290 kubelet.go:405] "Adding apiserver pod source" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.627547 4290 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.629417 4290 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.629432 4290 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.630071 4290 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.632522 4290 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.632536 4290 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.632643 4290 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.632912 4290 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.632954 4290 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.633064 4290 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.633081 4290 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.633089 4290 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.633097 4290 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.633106 4290 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.633116 4290 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.633124 4290 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.633132 4290 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.633141 4290 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.633150 4290 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.633164 4290 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.633178 4290 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.633207 4290 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 15:08:12.633577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.633216 4290 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 15:08:12.636486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.636467 4290 server.go:1263] "Started kubelet" Nov 03 15:08:12.636596 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.636545 4290 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 15:08:12.637661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.636692 4290 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 15:08:12.637661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.636814 4290 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 15:08:12.637661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.637260 4290 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 15:08:12.637033 ip-10-0-139-35 systemd[1]: Started Kubernetes Kubelet. Nov 03 15:08:12.638064 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.637682 4290 server.go:317] "Adding debug handlers to kubelet server" Nov 03 15:08:12.638833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.638816 4290 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 15:08:12.638833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.638828 4290 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.638893 4290 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:02:35 +0000 UTC" deadline="2028-05-25 13:21:46.780133406 +0000 UTC" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.638922 4290 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13654h13m34.141213982s" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.638939 4290 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.638954 4290 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.639046 4290 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.639982 4290 factory.go:55] Registering systemd factory Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.639999 4290 factory.go:223] Registration of the systemd container factory successfully Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640080 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfc1fa7c-1f46-4151-9375-cedcec2e8cfa" volumeName="kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovnkube-script-lib" seLinuxMountContext="" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640112 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfc1fa7c-1f46-4151-9375-cedcec2e8cfa" volumeName="kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-env-overrides" seLinuxMountContext="" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640127 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f997ae6c-4cdd-4def-ac45-e81c30896420" volumeName="kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret" seLinuxMountContext="" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640153 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="588c75cb-ff10-45c1-be72-625f6d9b9d9e" volumeName="kubernetes.io/empty-dir/588c75cb-ff10-45c1-be72-625f6d9b9d9e-tmp-dir" seLinuxMountContext="" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640164 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1129e2e-263b-48b8-a239-97d9164beca1" volumeName="kubernetes.io/empty-dir/a1129e2e-263b-48b8-a239-97d9164beca1-tmp" seLinuxMountContext="" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640175 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab771653-7ecd-4022-900f-c02f580bbb01" volumeName="kubernetes.io/configmap/ab771653-7ecd-4022-900f-c02f580bbb01-serviceca" seLinuxMountContext="" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640182 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c46a1f0f-4dc2-4b03-8269-d19a93977b56" volumeName="kubernetes.io/projected/c46a1f0f-4dc2-4b03-8269-d19a93977b56-kube-api-access-tgnz5" seLinuxMountContext="" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640189 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfc1fa7c-1f46-4151-9375-cedcec2e8cfa" volumeName="kubernetes.io/projected/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-kube-api-access-p6zql" seLinuxMountContext="" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640198 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfc1fa7c-1f46-4151-9375-cedcec2e8cfa" volumeName="kubernetes.io/secret/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640205 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="588c75cb-ff10-45c1-be72-625f6d9b9d9e" volumeName="kubernetes.io/projected/588c75cb-ff10-45c1-be72-625f6d9b9d9e-kube-api-access-dq5ct" seLinuxMountContext="" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640212 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c8010d3-c54b-48b1-b792-fbd8933863a2" volumeName="kubernetes.io/projected/5c8010d3-c54b-48b1-b792-fbd8933863a2-kube-api-access-5xpq6" seLinuxMountContext="" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640219 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d5de266-de42-430c-b364-d12951764b40" volumeName="kubernetes.io/projected/7d5de266-de42-430c-b364-d12951764b40-kube-api-access-xfpfv" seLinuxMountContext="" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640227 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b304a74d-882a-44e8-a696-4aff7665157c" volumeName="kubernetes.io/projected/b304a74d-882a-44e8-a696-4aff7665157c-kube-api-access-vdj7s" seLinuxMountContext="" Nov 03 15:08:12.640225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640233 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfc1fa7c-1f46-4151-9375-cedcec2e8cfa" volumeName="kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovnkube-config" seLinuxMountContext="" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640247 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c8010d3-c54b-48b1-b792-fbd8933863a2" volumeName="kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640257 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c8010d3-c54b-48b1-b792-fbd8933863a2" volumeName="kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-cni-binary-copy" seLinuxMountContext="" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640264 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1129e2e-263b-48b8-a239-97d9164beca1" volumeName="kubernetes.io/empty-dir/a1129e2e-263b-48b8-a239-97d9164beca1-etc-tuned" seLinuxMountContext="" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640271 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1129e2e-263b-48b8-a239-97d9164beca1" volumeName="kubernetes.io/projected/a1129e2e-263b-48b8-a239-97d9164beca1-kube-api-access-g7pzg" seLinuxMountContext="" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640278 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab771653-7ecd-4022-900f-c02f580bbb01" volumeName="kubernetes.io/projected/ab771653-7ecd-4022-900f-c02f580bbb01-kube-api-access-bwhk7" seLinuxMountContext="" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640285 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c46a1f0f-4dc2-4b03-8269-d19a93977b56" volumeName="kubernetes.io/configmap/c46a1f0f-4dc2-4b03-8269-d19a93977b56-cni-binary-copy" seLinuxMountContext="" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640291 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c46a1f0f-4dc2-4b03-8269-d19a93977b56" volumeName="kubernetes.io/configmap/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-daemon-config" seLinuxMountContext="" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640298 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4f18ec9-6d01-4a3d-9845-58b607a6225f" volumeName="kubernetes.io/configmap/c4f18ec9-6d01-4a3d-9845-58b607a6225f-konnectivity-ca" seLinuxMountContext="" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640305 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c8010d3-c54b-48b1-b792-fbd8933863a2" volumeName="kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640311 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d5de266-de42-430c-b364-d12951764b40" volumeName="kubernetes.io/configmap/7d5de266-de42-430c-b364-d12951764b40-iptables-alerter-script" seLinuxMountContext="" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640321 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d58285c-7e8d-4833-87a2-66ba8f398c8d" volumeName="kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st" seLinuxMountContext="" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640331 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6fc4400-11f3-42a8-afa3-68143e29eb10" volumeName="kubernetes.io/projected/a6fc4400-11f3-42a8-afa3-68143e29eb10-kube-api-access-8szqq" seLinuxMountContext="" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640347 4290 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4f18ec9-6d01-4a3d-9845-58b607a6225f" volumeName="kubernetes.io/secret/c4f18ec9-6d01-4a3d-9845-58b607a6225f-agent-certs" seLinuxMountContext="" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640354 4290 reconstruct.go:97] "Volume reconstruction finished" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640358 4290 reconciler.go:29] "Reconciler: start to sync state" Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640736 4290 factory.go:153] Registering CRI-O factory Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640753 4290 factory.go:223] Registration of the crio container factory successfully Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640839 4290 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640860 4290 factory.go:103] Registering Raw factory Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.640895 4290 manager.go:1199] Started watching for new ooms in manager Nov 03 15:08:12.642803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.641250 4290 manager.go:319] Starting recovery of all containers Nov 03 15:08:12.643988 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:08:12.643956 4290 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-35.us-west-1.compute.internal\" not found" Nov 03 15:08:12.668687 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.668643 4290 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 15:08:12.669973 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.669951 4290 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 15:08:12.669973 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.669974 4290 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 15:08:12.670125 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.669990 4290 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 15:08:12.670125 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:08:12.670021 4290 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 15:08:12.692331 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.692306 4290 generic.go:356] "Generic (PLEG): container finished" podID="fabcd339c8e2b42bb3c3bff7d173edec" containerID="cc03d66fb51322da4a1ff0f26ca5dc4080274054bd2a01a2289c9e445827ba69" exitCode=0 Nov 03 15:08:12.697597 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.697550 4290 generic.go:356] "Generic (PLEG): container finished" podID="5c8010d3-c54b-48b1-b792-fbd8933863a2" containerID="0b173cf78d8485ad2c56989a60237d03f57631cd0426cd16e3b0defbeac1b7bd" exitCode=0 Nov 03 15:08:12.697597 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.697594 4290 generic.go:356] "Generic (PLEG): container finished" podID="5c8010d3-c54b-48b1-b792-fbd8933863a2" containerID="629dbbcdcb7f8c786213edbcf6459ecab06787f0d246fce236377a60b9ae8535" exitCode=0 Nov 03 15:08:12.697597 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.697602 4290 generic.go:356] "Generic (PLEG): container finished" podID="5c8010d3-c54b-48b1-b792-fbd8933863a2" containerID="593a153b04748dbd119c651a47b349cd9f8f42fe9e1fc71eddc67d602150e9fa" exitCode=0 Nov 03 15:08:12.697809 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.697607 4290 generic.go:356] "Generic (PLEG): container finished" podID="5c8010d3-c54b-48b1-b792-fbd8933863a2" containerID="7e0f09c1342efa2fff76f2d807615c8608e8cf1570a57ee7be7096bed86b932c" exitCode=0 Nov 03 15:08:12.697809 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.697614 4290 generic.go:356] "Generic (PLEG): container finished" podID="5c8010d3-c54b-48b1-b792-fbd8933863a2" containerID="fb2e1970cfc9b97a5f9aafed1eef353206c4e8c56297add225fa581c60e141a2" exitCode=0 Nov 03 15:08:12.697809 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.697619 4290 generic.go:356] "Generic (PLEG): container finished" podID="5c8010d3-c54b-48b1-b792-fbd8933863a2" containerID="853dbc9869c3fba680c4b1bb079a4823316022cc2adccfdcd5a73e8d018d30f6" exitCode=0 Nov 03 15:08:12.704041 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.704023 4290 manager.go:324] Recovery completed Nov 03 15:08:12.715288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.715274 4290 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:08:12.716253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.716237 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:08:12.716298 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.716272 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:08:12.716298 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.716284 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:08:12.717429 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.717415 4290 cpu_manager.go:226] "Starting" policy="none" Nov 03 15:08:12.717429 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.717426 4290 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 15:08:12.717429 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.717428 4290 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:08:12.717533 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.717441 4290 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 15:08:12.717570 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.717551 4290 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 15:08:12.717602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.717562 4290 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 15:08:12.717602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.717583 4290 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 15:08:12.717602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.717590 4290 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 15:08:12.717602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.717595 4290 policy_none.go:50] "Start" Nov 03 15:08:12.717602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.717602 4290 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 15:08:12.717783 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.717611 4290 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 15:08:12.718225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.718197 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:08:12.718271 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.718245 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:08:12.718271 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.718260 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:08:12.720308 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.720295 4290 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 15:08:12.720352 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.720311 4290 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 15:08:12.720352 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.720317 4290 policy_none.go:44] "Start" Nov 03 15:08:12.724083 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.724060 4290 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 15:08:12.726690 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.726678 4290 manager.go:341] "Starting Device Plugin manager" Nov 03 15:08:12.726759 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:08:12.726723 4290 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 15:08:12.726759 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.726733 4290 server.go:87] "Starting device plugin registration server" Nov 03 15:08:12.727053 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.727039 4290 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 15:08:12.727091 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.727056 4290 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 15:08:12.727167 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.727145 4290 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 15:08:12.727458 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.727411 4290 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 15:08:12.727546 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.727466 4290 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 15:08:12.727546 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.727478 4290 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 15:08:12.727656 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.727523 4290 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T15:08:12.727450602Z","UUID":"73a155c5-02bd-47a4-b654-5611b14d069a","Handler":null,"Name":"","Endpoint":""} Nov 03 15:08:12.728476 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:08:12.728448 4290 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 15:08:12.728549 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:08:12.728485 4290 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-35.us-west-1.compute.internal\" not found" Nov 03 15:08:12.728877 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.728862 4290 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 15:08:12.728961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.728884 4290 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 15:08:12.770808 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.770285 4290 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal"] Nov 03 15:08:12.770902 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.770877 4290 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:08:12.771723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.771691 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:08:12.771766 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.771735 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:08:12.771766 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.771745 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:08:12.771946 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.771937 4290 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:08:12.772070 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.772054 4290 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:08:12.772664 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.772647 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:08:12.772778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.772670 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:08:12.772778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.772680 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:08:12.772778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.772726 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:08:12.772778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.772752 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:08:12.772778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.772763 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:08:12.772945 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:08:12.772876 4290 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-35.us-west-1.compute.internal\" not found" node="ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:12.772945 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.772889 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" event={"ID":"fabcd339c8e2b42bb3c3bff7d173edec","Type":"ContainerStarted","Data":"ed4e592e47b74b7af45d88c66247d0fa69fc6aa9c3290bcb7ad4ea8a68999e15"} Nov 03 15:08:12.772945 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.772924 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" event={"ID":"fabcd339c8e2b42bb3c3bff7d173edec","Type":"ContainerDied","Data":"cc03d66fb51322da4a1ff0f26ca5dc4080274054bd2a01a2289c9e445827ba69"} Nov 03 15:08:12.772945 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.772939 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" event={"ID":"fabcd339c8e2b42bb3c3bff7d173edec","Type":"ContainerStarted","Data":"6f735fd43670028f25e285734a6ac5bd76a2753e3c11ab0687aa1864def51645"} Nov 03 15:08:12.773062 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.772968 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" event={"ID":"d955cef443a4983dada78f8e0a328c65","Type":"ContainerStarted","Data":"5f799f874ac83e1fa6521ec1dd628a31a0bd9d1a7c8a184670798a33cf1ac513"} Nov 03 15:08:12.773062 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.772980 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" event={"ID":"d955cef443a4983dada78f8e0a328c65","Type":"ContainerStarted","Data":"63a79073cf5f69c69bc4d1d8f29085e892cbe545d9f536ec9217b224fe6aafcb"} Nov 03 15:08:12.773062 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.772992 4290 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:08:12.773829 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.773780 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:08:12.773829 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.773806 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:08:12.773829 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.773815 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:08:12.773962 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:08:12.773933 4290 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-35.us-west-1.compute.internal\" not found" node="ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:12.827153 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.827131 4290 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:08:12.827910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.827895 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:08:12.827960 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.827919 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:08:12.827960 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.827929 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:08:12.827960 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.827942 4290 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:12.843202 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.843183 4290 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 15:08:12.890897 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.890881 4290 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:08:12.890983 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.890967 4290 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:08:12.892747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.892733 4290 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:08:12.928256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.928239 4290 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:08:12.940529 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.940508 4290 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:12.944055 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.944038 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d955cef443a4983dada78f8e0a328c65-config\") pod \"kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"d955cef443a4983dada78f8e0a328c65\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:12.944144 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.944066 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fabcd339c8e2b42bb3c3bff7d173edec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"fabcd339c8e2b42bb3c3bff7d173edec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:12.944144 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.944092 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fabcd339c8e2b42bb3c3bff7d173edec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"fabcd339c8e2b42bb3c3bff7d173edec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:12.980820 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.980801 4290 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:12.980885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.980857 4290 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:12.981377 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.981363 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeNotReady" Nov 03 15:08:12.981444 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.981389 4290 setters.go:546] "Node became not ready" node="ip-10-0-139-35.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T15:08:12Z","lastTransitionTime":"2026-11-03T15:08:12Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 15:08:12.993620 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:12.993600 4290 nodeinfomanager.go:471] existing CSINode "ip-10-0-139-35.us-west-1.compute.internal" is owned by different node (oldNodeID="56b7ad6c-084b-462a-8b27-a7cbfe5582b2", newNodeID=""), cleaning up... Nov 03 15:08:13.003823 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.003804 4290 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 15:08:13.003872 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:08:13.003843 4290 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:13.003872 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.003855 4290 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:13.044363 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.044313 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d955cef443a4983dada78f8e0a328c65-config\") pod \"kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"d955cef443a4983dada78f8e0a328c65\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:13.044363 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.044337 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fabcd339c8e2b42bb3c3bff7d173edec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"fabcd339c8e2b42bb3c3bff7d173edec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:13.044363 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.044353 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fabcd339c8e2b42bb3c3bff7d173edec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"fabcd339c8e2b42bb3c3bff7d173edec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:13.044502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.044399 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fabcd339c8e2b42bb3c3bff7d173edec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"fabcd339c8e2b42bb3c3bff7d173edec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:13.044502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.044415 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d955cef443a4983dada78f8e0a328c65-config\") pod \"kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"d955cef443a4983dada78f8e0a328c65\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:13.044502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.044422 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fabcd339c8e2b42bb3c3bff7d173edec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal\" (UID: \"fabcd339c8e2b42bb3c3bff7d173edec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:13.063386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.063364 4290 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 15:08:13.063447 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:08:13.063404 4290 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:13.628354 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.628327 4290 apiserver.go:52] "Watching apiserver" Nov 03 15:08:13.703278 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.703255 4290 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:13.703390 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.703380 4290 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:13.731937 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.731916 4290 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 15:08:13.732266 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.732250 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-dtkj5","kube-system/konnectivity-agent-q5rxt","kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7","openshift-network-diagnostics/network-check-target-qwzv7","openshift-ovn-kubernetes/ovnkube-node-z7xfz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-fkx68","openshift-network-operator/iptables-alerter-rq6pc","openshift-cluster-node-tuning-operator/tuned-qx6fv","kube-system/global-pull-secret-syncer-hwchb","openshift-dns/node-resolver-cmv4d","openshift-image-registry/node-ca-7fz2g","openshift-multus/multus-additional-cni-plugins-zkw2x"] Nov 03 15:08:13.739004 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.738977 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:08:13.747758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.747734 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-sys\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.747843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.747766 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdj7s\" (UniqueName: \"kubernetes.io/projected/b304a74d-882a-44e8-a696-4aff7665157c-kube-api-access-vdj7s\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.747843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.747782 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-systemd\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.747843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.747804 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-etc-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.747843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.747828 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.748027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.747891 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.748027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.747965 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovn-node-metrics-cert\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.748027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748000 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovnkube-script-lib\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.748143 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748106 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-device-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.748173 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748155 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-var-lib-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.748201 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748175 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwhk7\" (UniqueName: \"kubernetes.io/projected/ab771653-7ecd-4022-900f-c02f580bbb01-kube-api-access-bwhk7\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:08:13.748229 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748192 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.748314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748299 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4st\" (UniqueName: \"kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st\") pod \"network-check-target-qwzv7\" (UID: \"9d58285c-7e8d-4833-87a2-66ba8f398c8d\") " pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:08:13.748365 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748352 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d5de266-de42-430c-b364-d12951764b40-iptables-alerter-script\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:08:13.748413 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748375 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-var-lib-kubelet\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.748413 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748393 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1129e2e-263b-48b8-a239-97d9164beca1-etc-tuned\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.748413 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748410 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.748549 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748427 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-slash\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.748549 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748441 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-run-netns\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.748549 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748465 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1129e2e-263b-48b8-a239-97d9164beca1-etc-tuned\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.748549 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748529 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.748739 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748572 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.748739 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748624 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.748739 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748646 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-node-log\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.748739 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748692 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-run-ovn-kubernetes\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.748930 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748754 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d5de266-de42-430c-b364-d12951764b40-host-slash\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:08:13.748930 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748803 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8szqq\" (UniqueName: \"kubernetes.io/projected/a6fc4400-11f3-42a8-afa3-68143e29eb10-kube-api-access-8szqq\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:08:13.748930 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748843 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-kubernetes\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.748930 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748875 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-run\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.749099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748961 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-registration-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.749099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.748988 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-env-overrides\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.749099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749039 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-modprobe-d\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.749099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749055 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-systemd\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.749099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749069 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-sys-fs\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.749099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749081 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-cni-bin\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.749099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749101 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:08:13.749315 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749116 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysctl-d\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.749315 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749154 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-cnibin\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.749315 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749182 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xpq6\" (UniqueName: \"kubernetes.io/projected/5c8010d3-c54b-48b1-b792-fbd8933863a2-kube-api-access-5xpq6\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.749315 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749273 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-kubelet\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.749315 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749289 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-cni-netd\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.749315 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749305 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1129e2e-263b-48b8-a239-97d9164beca1-tmp\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.749474 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749319 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-ovn\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.749474 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749353 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-log-socket\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.749474 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749403 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1129e2e-263b-48b8-a239-97d9164beca1-tmp\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.749474 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749419 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovnkube-config\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.749474 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749436 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-systemd-units\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.749602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749487 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4f18ec9-6d01-4a3d-9845-58b607a6225f-agent-certs\") pod \"konnectivity-agent-q5rxt\" (UID: \"c4f18ec9-6d01-4a3d-9845-58b607a6225f\") " pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:08:13.749602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749514 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysctl-conf\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.749602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749555 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-system-cni-dir\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.749602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749573 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-socket-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.749602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749588 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-etc-selinux\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.749764 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749613 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6zql\" (UniqueName: \"kubernetes.io/projected/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-kube-api-access-p6zql\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.749764 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749642 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4f18ec9-6d01-4a3d-9845-58b607a6225f-konnectivity-ca\") pod \"konnectivity-agent-q5rxt\" (UID: \"c4f18ec9-6d01-4a3d-9845-58b607a6225f\") " pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:08:13.749764 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749732 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysconfig\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.749893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749774 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7pzg\" (UniqueName: \"kubernetes.io/projected/a1129e2e-263b-48b8-a239-97d9164beca1-kube-api-access-g7pzg\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.749893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749792 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab771653-7ecd-4022-900f-c02f580bbb01-host\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:08:13.749893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749848 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab771653-7ecd-4022-900f-c02f580bbb01-serviceca\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:08:13.749967 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749891 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfpfv\" (UniqueName: \"kubernetes.io/projected/7d5de266-de42-430c-b364-d12951764b40-kube-api-access-xfpfv\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:08:13.749967 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749908 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-lib-modules\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.749967 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.749923 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-host\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.763080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.763059 4290 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 15:08:13.763163 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:08:13.763099 4290 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:13.763715 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.763686 4290 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 15:08:13.763780 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:08:13.763730 4290 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-35.us-west-1.compute.internal" Nov 03 15:08:13.785258 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.785235 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kdt56\"" Nov 03 15:08:13.785258 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.785249 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jpxkd\"" Nov 03 15:08:13.785920 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.785905 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 15:08:13.785920 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.785905 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 15:08:13.786061 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.786025 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 15:08:13.786119 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.786028 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 15:08:13.786166 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.786140 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 15:08:13.786166 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.786163 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 15:08:13.786279 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.786260 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 15:08:13.786943 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.786924 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 15:08:13.787554 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.787537 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-975kv\"" Nov 03 15:08:13.787639 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.787571 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 15:08:13.787727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.787672 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qsch2\"" Nov 03 15:08:13.787727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.787674 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fsh2r\"" Nov 03 15:08:13.788877 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.788860 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.788947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.788878 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.789981 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.789965 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6cpt6\"" Nov 03 15:08:13.790101 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.790080 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab771653-7ecd-4022-900f-c02f580bbb01-serviceca\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:08:13.790390 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.790377 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 15:08:13.791179 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.791166 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cjtnz\"" Nov 03 15:08:13.791347 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.791333 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 15:08:13.791409 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.791358 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z77jb\"" Nov 03 15:08:13.791591 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.791575 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 15:08:13.791712 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.791687 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4wckw\"" Nov 03 15:08:13.791940 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.791926 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dfz9p\"" Nov 03 15:08:13.792013 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.791945 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cblvv\"" Nov 03 15:08:13.792204 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.792190 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qbvnh\"" Nov 03 15:08:13.792341 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.792330 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 15:08:13.798509 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.798496 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovn-node-metrics-cert\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.799769 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.799756 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4f18ec9-6d01-4a3d-9845-58b607a6225f-agent-certs\") pod \"konnectivity-agent-q5rxt\" (UID: \"c4f18ec9-6d01-4a3d-9845-58b607a6225f\") " pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:08:13.815055 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.815031 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7pzg\" (UniqueName: \"kubernetes.io/projected/a1129e2e-263b-48b8-a239-97d9164beca1-kube-api-access-g7pzg\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.815121 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.815032 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwhk7\" (UniqueName: \"kubernetes.io/projected/ab771653-7ecd-4022-900f-c02f580bbb01-kube-api-access-bwhk7\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:08:13.839429 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.839404 4290 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 15:08:13.850971 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.850947 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-systemd\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851042 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.850979 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-etc-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851042 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851002 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851042 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851027 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851042 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851034 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-etc-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851036 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-systemd\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851054 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-device-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.851217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851060 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851078 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851081 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-netns\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.851217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851120 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-device-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.851217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851124 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-var-lib-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851158 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f997ae6c-4cdd-4def-ac45-e81c30896420-dbus\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:08:13.851217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851166 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-var-lib-openvswitch\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851186 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-system-cni-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.851217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851224 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-kubelet\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851251 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-hostroot\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851278 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-var-lib-kubelet\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851307 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-slash\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851337 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-run-netns\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851370 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851382 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851448 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851437 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-slash\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851425 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851478 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851493 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851384 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-run-netns\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851512 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851403 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-var-lib-kubelet\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851539 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851565 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851585 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851613 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851611 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851596 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgnz5\" (UniqueName: \"kubernetes.io/projected/c46a1f0f-4dc2-4b03-8269-d19a93977b56-kube-api-access-tgnz5\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851653 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851662 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851715 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851726 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851745 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851750 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-node-log\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851766 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851775 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851693 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-node-log\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851793 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851810 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-run-ovn-kubernetes\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.851819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851837 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d5de266-de42-430c-b364-d12951764b40-host-slash\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851856 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-kubernetes\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851895 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-kubernetes\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851896 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d5de266-de42-430c-b364-d12951764b40-host-slash\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851906 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-run-ovn-kubernetes\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851916 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-run\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851783 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851952 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-cnibin\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.851994 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-registration-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852015 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5ct\" (UniqueName: \"kubernetes.io/projected/588c75cb-ff10-45c1-be72-625f6d9b9d9e-kube-api-access-dq5ct\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852040 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-run\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852092 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-modprobe-d\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852088 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-registration-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852113 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-systemd\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852176 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852235 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-systemd\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852236 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-sys-fs\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852276 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-sys-fs\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852281 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-modprobe-d\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852320 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f997ae6c-4cdd-4def-ac45-e81c30896420-kubelet-config\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852338 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/588c75cb-ff10-45c1-be72-625f6d9b9d9e-tmp-dir\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852359 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-cni-bin\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852391 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-cni-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852425 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-cni-bin\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852456 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852465 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/588c75cb-ff10-45c1-be72-625f6d9b9d9e-tmp-dir\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852494 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysctl-d\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852510 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-cnibin\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852611 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-etc-kubernetes\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852628 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-cnibin\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852666 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-kubelet\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852694 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysctl-d\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.852688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852693 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-cni-netd\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852738 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c46a1f0f-4dc2-4b03-8269-d19a93977b56-cni-binary-copy\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852744 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-cni-netd\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852767 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-host-kubelet\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852772 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-conf-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852803 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-multus-certs\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852806 4290 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852824 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-ovn\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852864 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-run-ovn\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852870 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-log-socket\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852915 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852949 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c46a1f0f-4dc2-4b03-8269-d19a93977b56-cni-binary-copy\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852954 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-systemd-units\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.852958 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-log-socket\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853034 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-systemd-units\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853045 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysctl-conf\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853068 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-system-cni-dir\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853084 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-socket-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853097 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-etc-selinux\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853115 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-k8s-cni-cncf-io\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853140 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysconfig\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853160 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysctl-conf\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853198 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-socket-dir\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853215 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c8010d3-c54b-48b1-b792-fbd8933863a2-system-cni-dir\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853232 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab771653-7ecd-4022-900f-c02f580bbb01-host\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853243 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f997ae6c-4cdd-4def-ac45-e81c30896420-original-pull-secret\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853295 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-socket-dir-parent\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853303 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab771653-7ecd-4022-900f-c02f580bbb01-host\") pod \"node-ca-7fz2g\" (UID: \"ab771653-7ecd-4022-900f-c02f580bbb01\") " pod="openshift-image-registry/node-ca-7fz2g" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853274 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b304a74d-882a-44e8-a696-4aff7665157c-etc-selinux\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853338 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-cni-multus\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853397 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-etc-sysconfig\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853432 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-lib-modules\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853452 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-host\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853466 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-daemon-config\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853482 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-sys\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853507 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-cni-bin\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853514 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-host\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853521 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-lib-modules\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853530 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/588c75cb-ff10-45c1-be72-625f6d9b9d9e-hosts-file\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853536 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1129e2e-263b-48b8-a239-97d9164beca1-sys\") pod \"tuned-qx6fv\" (UID: \"a1129e2e-263b-48b8-a239-97d9164beca1\") " pod="openshift-cluster-node-tuning-operator/tuned-qx6fv" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853636 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-daemon-config\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.853859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.853672 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xpq6\" (UniqueName: \"kubernetes.io/projected/5c8010d3-c54b-48b1-b792-fbd8933863a2-kube-api-access-5xpq6\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.855002 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.854148 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfpfv\" (UniqueName: \"kubernetes.io/projected/7d5de266-de42-430c-b364-d12951764b40-kube-api-access-xfpfv\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:08:13.855222 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.855191 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6fc4400-11f3-42a8-afa3-68143e29eb10-metrics-certs\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:08:13.855435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.855416 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6zql\" (UniqueName: \"kubernetes.io/projected/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-kube-api-access-p6zql\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.855505 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.855416 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8szqq\" (UniqueName: \"kubernetes.io/projected/a6fc4400-11f3-42a8-afa3-68143e29eb10-kube-api-access-8szqq\") pod \"network-metrics-daemon-fkx68\" (UID: \"a6fc4400-11f3-42a8-afa3-68143e29eb10\") " pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:08:13.855589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.855570 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdj7s\" (UniqueName: \"kubernetes.io/projected/b304a74d-882a-44e8-a696-4aff7665157c-kube-api-access-vdj7s\") pod \"aws-ebs-csi-driver-node-cpsx7\" (UID: \"b304a74d-882a-44e8-a696-4aff7665157c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cpsx7" Nov 03 15:08:13.855729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.855691 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m4st\" (UniqueName: \"kubernetes.io/projected/9d58285c-7e8d-4833-87a2-66ba8f398c8d-kube-api-access-9m4st\") pod \"network-check-target-qwzv7\" (UID: \"9d58285c-7e8d-4833-87a2-66ba8f398c8d\") " pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:08:13.858561 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.858538 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovnkube-script-lib\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.858607 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.858539 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c8010d3-c54b-48b1-b792-fbd8933863a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkw2x\" (UID: \"5c8010d3-c54b-48b1-b792-fbd8933863a2\") " pod="openshift-multus/multus-additional-cni-plugins-zkw2x" Nov 03 15:08:13.858607 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.858582 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d5de266-de42-430c-b364-d12951764b40-iptables-alerter-script\") pod \"iptables-alerter-rq6pc\" (UID: \"7d5de266-de42-430c-b364-d12951764b40\") " pod="openshift-network-operator/iptables-alerter-rq6pc" Nov 03 15:08:13.859745 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.859726 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-ovnkube-config\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.859834 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.859799 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dfc1fa7c-1f46-4151-9375-cedcec2e8cfa-env-overrides\") pod \"ovnkube-node-z7xfz\" (UID: \"dfc1fa7c-1f46-4151-9375-cedcec2e8cfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:13.859885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.859873 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4f18ec9-6d01-4a3d-9845-58b607a6225f-konnectivity-ca\") pod \"konnectivity-agent-q5rxt\" (UID: \"c4f18ec9-6d01-4a3d-9845-58b607a6225f\") " pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:08:13.911522 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.911470 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgnz5\" (UniqueName: \"kubernetes.io/projected/c46a1f0f-4dc2-4b03-8269-d19a93977b56-kube-api-access-tgnz5\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.911522 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.911511 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5ct\" (UniqueName: \"kubernetes.io/projected/588c75cb-ff10-45c1-be72-625f6d9b9d9e-kube-api-access-dq5ct\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:08:13.954419 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954393 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-etc-kubernetes\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954419 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954420 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-conf-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954433 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-multus-certs\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954452 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-k8s-cni-cncf-io\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954466 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-socket-dir-parent\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954500 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-k8s-cni-cncf-io\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954499 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-conf-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954518 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-etc-kubernetes\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954516 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-multus-certs\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954524 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-cni-multus\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954552 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-cni-multus\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954566 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-socket-dir-parent\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954570 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-cni-bin\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954594 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-cni-bin\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954613 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/588c75cb-ff10-45c1-be72-625f6d9b9d9e-hosts-file\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954642 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-netns\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954667 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/588c75cb-ff10-45c1-be72-625f6d9b9d9e-hosts-file\") pod \"node-resolver-cmv4d\" (UID: \"588c75cb-ff10-45c1-be72-625f6d9b9d9e\") " pod="openshift-dns/node-resolver-cmv4d" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954668 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f997ae6c-4cdd-4def-ac45-e81c30896420-dbus\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954691 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-system-cni-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954728 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-kubelet\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954736 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f997ae6c-4cdd-4def-ac45-e81c30896420-dbus\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954747 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-run-netns\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954751 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-hostroot\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954773 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-system-cni-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954789 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-host-var-lib-kubelet\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954800 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-cnibin\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954819 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f997ae6c-4cdd-4def-ac45-e81c30896420-kubelet-config\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954831 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-hostroot\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954834 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-cni-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954871 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-cnibin\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954898 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c46a1f0f-4dc2-4b03-8269-d19a93977b56-multus-cni-dir\") pod \"multus-dtkj5\" (UID: \"c46a1f0f-4dc2-4b03-8269-d19a93977b56\") " pod="openshift-multus/multus-dtkj5" Nov 03 15:08:13.954947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:13.954900 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f997ae6c-4cdd-4def-ac45-e81c30896420-kubelet-config\") pod \"global-pull-secret-syncer-hwchb\" (UID: \"f997ae6c-4cdd-4def-ac45-e81c30896420\") " pod="kube-system/global-pull-secret-syncer-hwchb" Nov 03 15:08:14.039292 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:14.039263 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkx68" Nov 03 15:08:14.408912 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:14.408844 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fkx68"] Nov 03 15:08:14.412946 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:14.412909 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6fc4400_11f3_42a8_afa3_68143e29eb10.slice/crio-416b8f6e3636bf449c6b934a63f4d68b96764fc10fc01e7ae3c5fbfe50762765 WatchSource:0}: Error finding container 416b8f6e3636bf449c6b934a63f4d68b96764fc10fc01e7ae3c5fbfe50762765: Status 404 returned error can't find the container with id 416b8f6e3636bf449c6b934a63f4d68b96764fc10fc01e7ae3c5fbfe50762765 Nov 03 15:08:14.415487 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:14.415470 4290 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:08:14.706691 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:14.706613 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fkx68" event={"ID":"a6fc4400-11f3-42a8-afa3-68143e29eb10","Type":"ContainerStarted","Data":"416b8f6e3636bf449c6b934a63f4d68b96764fc10fc01e7ae3c5fbfe50762765"} Nov 03 15:08:16.713605 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:16.713572 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fkx68" event={"ID":"a6fc4400-11f3-42a8-afa3-68143e29eb10","Type":"ContainerStarted","Data":"b74f6d10f0282439ab957aa57eca97ed950272f30b1b68eba8edc818ef78d0ae"} Nov 03 15:08:16.713978 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:16.713618 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fkx68" event={"ID":"a6fc4400-11f3-42a8-afa3-68143e29eb10","Type":"ContainerStarted","Data":"deef9180f69e17d7646d50f9f78c26991745849e1388d11c4220bfac6acfc8ac"} Nov 03 15:08:19.324089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:19.324061 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:20.590757 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:20.590726 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:20.591130 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:20.590930 4290 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 15:08:23.223343 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:23.223315 4290 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-35.us-west-1.compute.internal" event="NodeReady" Nov 03 15:08:23.435014 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:23.434987 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:08:24.229827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.229802 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7xfz" Nov 03 15:08:24.391884 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.391854 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kfw9b"] Nov 03 15:08:24.409530 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.409504 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6d2mt"] Nov 03 15:08:24.409675 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.409623 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kfw9b" Nov 03 15:08:24.423513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.423489 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6fvtf"] Nov 03 15:08:24.423623 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.423607 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:24.435639 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.435617 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-2drvj"] Nov 03 15:08:24.435967 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.435949 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6fvtf" Nov 03 15:08:24.457715 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.457682 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p8qkv"] Nov 03 15:08:24.457908 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.457847 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-2drvj" Nov 03 15:08:24.471751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.471725 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kfw9b"] Nov 03 15:08:24.471838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.471766 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6d2mt"] Nov 03 15:08:24.471838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.471793 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6fvtf"] Nov 03 15:08:24.471838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.471811 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.483014 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.482966 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-2drvj"] Nov 03 15:08:24.504283 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.504258 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 15:08:24.513168 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.513151 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6-crio-socket\") pod \"insights-runtime-extractor-p8qkv\" (UID: \"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6\") " pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.513232 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.513181 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6-metrics-tls\") pod \"dns-default-6d2mt\" (UID: \"c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6\") " pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:24.513232 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.513203 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p8qkv\" (UID: \"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6\") " pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.513300 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.513260 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d582038-d053-4b49-b302-0b7188666463-tmp\") pod \"downloads-6dd789c8b7-2drvj\" (UID: \"6d582038-d053-4b49-b302-0b7188666463\") " pod="openshift-console/downloads-6dd789c8b7-2drvj" Nov 03 15:08:24.513300 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.513289 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9846b382-908f-452b-afb6-083dfc331d62-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-kfw9b\" (UID: \"9846b382-908f-452b-afb6-083dfc331d62\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kfw9b" Nov 03 15:08:24.513372 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.513314 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6pvz\" (UniqueName: \"kubernetes.io/projected/6d582038-d053-4b49-b302-0b7188666463-kube-api-access-g6pvz\") pod \"downloads-6dd789c8b7-2drvj\" (UID: \"6d582038-d053-4b49-b302-0b7188666463\") " pod="openshift-console/downloads-6dd789c8b7-2drvj" Nov 03 15:08:24.513372 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.513357 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcmwz\" (UniqueName: \"kubernetes.io/projected/84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6-kube-api-access-tcmwz\") pod \"insights-runtime-extractor-p8qkv\" (UID: \"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6\") " pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.513473 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.513376 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6-config-volume\") pod \"dns-default-6d2mt\" (UID: \"c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6\") " pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:24.513473 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.513423 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6-tmp-dir\") pod \"dns-default-6d2mt\" (UID: \"c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6\") " pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:24.513473 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.513459 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc2sv\" (UniqueName: \"kubernetes.io/projected/c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6-kube-api-access-rc2sv\") pod \"dns-default-6d2mt\" (UID: \"c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6\") " pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:24.513619 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.513488 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1381acc6-37dc-484b-be29-9f9c617af63d-cert\") pod \"ingress-canary-6fvtf\" (UID: \"1381acc6-37dc-484b-be29-9f9c617af63d\") " pod="openshift-ingress-canary/ingress-canary-6fvtf" Nov 03 15:08:24.513619 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.513521 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p8qkv\" (UID: \"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6\") " pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.513619 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.513612 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snxcl\" (UniqueName: \"kubernetes.io/projected/1381acc6-37dc-484b-be29-9f9c617af63d-kube-api-access-snxcl\") pod \"ingress-canary-6fvtf\" (UID: \"1381acc6-37dc-484b-be29-9f9c617af63d\") " pod="openshift-ingress-canary/ingress-canary-6fvtf" Nov 03 15:08:24.513765 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.513639 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6-data-volume\") pod \"insights-runtime-extractor-p8qkv\" (UID: \"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6\") " pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.518387 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.518371 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gzlvm\"" Nov 03 15:08:24.518475 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.518406 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zncwb\"" Nov 03 15:08:24.518475 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.518430 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 15:08:24.531376 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.531356 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 15:08:24.531749 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.531726 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 15:08:24.531749 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.531728 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 15:08:24.531885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.531758 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 15:08:24.531885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.531770 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 15:08:24.531885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.531771 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 15:08:24.531885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.531798 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 15:08:24.531885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.531799 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 15:08:24.545827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.545803 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p8qkv"] Nov 03 15:08:24.588061 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.588022 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ncdvq\"" Nov 03 15:08:24.588164 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.588068 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g9729\"" Nov 03 15:08:24.588288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.588273 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 15:08:24.588341 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.588291 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 15:08:24.588382 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.588348 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-v25cx\"" Nov 03 15:08:24.614361 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614336 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p8qkv\" (UID: \"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6\") " pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.614464 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614366 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d582038-d053-4b49-b302-0b7188666463-tmp\") pod \"downloads-6dd789c8b7-2drvj\" (UID: \"6d582038-d053-4b49-b302-0b7188666463\") " pod="openshift-console/downloads-6dd789c8b7-2drvj" Nov 03 15:08:24.614464 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614383 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9846b382-908f-452b-afb6-083dfc331d62-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-kfw9b\" (UID: \"9846b382-908f-452b-afb6-083dfc331d62\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kfw9b" Nov 03 15:08:24.614464 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614407 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6pvz\" (UniqueName: \"kubernetes.io/projected/6d582038-d053-4b49-b302-0b7188666463-kube-api-access-g6pvz\") pod \"downloads-6dd789c8b7-2drvj\" (UID: \"6d582038-d053-4b49-b302-0b7188666463\") " pod="openshift-console/downloads-6dd789c8b7-2drvj" Nov 03 15:08:24.614464 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614432 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcmwz\" (UniqueName: \"kubernetes.io/projected/84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6-kube-api-access-tcmwz\") pod \"insights-runtime-extractor-p8qkv\" (UID: \"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6\") " pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.614464 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614456 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6-config-volume\") pod \"dns-default-6d2mt\" (UID: \"c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6\") " pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:24.614762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614482 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6-tmp-dir\") pod \"dns-default-6d2mt\" (UID: \"c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6\") " pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:24.614762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614504 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc2sv\" (UniqueName: \"kubernetes.io/projected/c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6-kube-api-access-rc2sv\") pod \"dns-default-6d2mt\" (UID: \"c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6\") " pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:24.614762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614535 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1381acc6-37dc-484b-be29-9f9c617af63d-cert\") pod \"ingress-canary-6fvtf\" (UID: \"1381acc6-37dc-484b-be29-9f9c617af63d\") " pod="openshift-ingress-canary/ingress-canary-6fvtf" Nov 03 15:08:24.614762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614567 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p8qkv\" (UID: \"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6\") " pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.614762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614607 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snxcl\" (UniqueName: \"kubernetes.io/projected/1381acc6-37dc-484b-be29-9f9c617af63d-kube-api-access-snxcl\") pod \"ingress-canary-6fvtf\" (UID: \"1381acc6-37dc-484b-be29-9f9c617af63d\") " pod="openshift-ingress-canary/ingress-canary-6fvtf" Nov 03 15:08:24.614762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614631 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6-data-volume\") pod \"insights-runtime-extractor-p8qkv\" (UID: \"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6\") " pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.614762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614658 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6-crio-socket\") pod \"insights-runtime-extractor-p8qkv\" (UID: \"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6\") " pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.614762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614727 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6-metrics-tls\") pod \"dns-default-6d2mt\" (UID: \"c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6\") " pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:24.615175 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614933 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d582038-d053-4b49-b302-0b7188666463-tmp\") pod \"downloads-6dd789c8b7-2drvj\" (UID: \"6d582038-d053-4b49-b302-0b7188666463\") " pod="openshift-console/downloads-6dd789c8b7-2drvj" Nov 03 15:08:24.615175 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.614947 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6-tmp-dir\") pod \"dns-default-6d2mt\" (UID: \"c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6\") " pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:24.615175 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.615037 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p8qkv\" (UID: \"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6\") " pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.615323 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.615198 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6-config-volume\") pod \"dns-default-6d2mt\" (UID: \"c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6\") " pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:24.615455 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.615426 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6-data-volume\") pod \"insights-runtime-extractor-p8qkv\" (UID: \"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6\") " pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.615637 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.615620 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6-crio-socket\") pod \"insights-runtime-extractor-p8qkv\" (UID: \"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6\") " pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.617333 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.617308 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9846b382-908f-452b-afb6-083dfc331d62-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-kfw9b\" (UID: \"9846b382-908f-452b-afb6-083dfc331d62\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kfw9b" Nov 03 15:08:24.617922 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.617901 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1381acc6-37dc-484b-be29-9f9c617af63d-cert\") pod \"ingress-canary-6fvtf\" (UID: \"1381acc6-37dc-484b-be29-9f9c617af63d\") " pod="openshift-ingress-canary/ingress-canary-6fvtf" Nov 03 15:08:24.617997 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.617904 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6-metrics-tls\") pod \"dns-default-6d2mt\" (UID: \"c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6\") " pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:24.617997 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.617956 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p8qkv\" (UID: \"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6\") " pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.676254 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.676225 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6pvz\" (UniqueName: \"kubernetes.io/projected/6d582038-d053-4b49-b302-0b7188666463-kube-api-access-g6pvz\") pod \"downloads-6dd789c8b7-2drvj\" (UID: \"6d582038-d053-4b49-b302-0b7188666463\") " pod="openshift-console/downloads-6dd789c8b7-2drvj" Nov 03 15:08:24.677228 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.677201 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcmwz\" (UniqueName: \"kubernetes.io/projected/84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6-kube-api-access-tcmwz\") pod \"insights-runtime-extractor-p8qkv\" (UID: \"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6\") " pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:24.677333 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.677202 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snxcl\" (UniqueName: \"kubernetes.io/projected/1381acc6-37dc-484b-be29-9f9c617af63d-kube-api-access-snxcl\") pod \"ingress-canary-6fvtf\" (UID: \"1381acc6-37dc-484b-be29-9f9c617af63d\") " pod="openshift-ingress-canary/ingress-canary-6fvtf" Nov 03 15:08:24.677333 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.677294 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc2sv\" (UniqueName: \"kubernetes.io/projected/c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6-kube-api-access-rc2sv\") pod \"dns-default-6d2mt\" (UID: \"c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6\") " pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:24.736489 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.736436 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kfw9b" Nov 03 15:08:24.772546 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.772527 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:24.792332 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.792287 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-2drvj" Nov 03 15:08:24.811021 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.810997 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6fvtf" Nov 03 15:08:24.847106 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:24.847075 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p8qkv" Nov 03 15:08:25.123662 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:25.123637 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kfw9b"] Nov 03 15:08:25.126610 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:25.126580 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9846b382_908f_452b_afb6_083dfc331d62.slice/crio-56564c56aa716d3c3d38a8e6ad5553476643b7fe6c89e1eef0cf82da1592a6b2 WatchSource:0}: Error finding container 56564c56aa716d3c3d38a8e6ad5553476643b7fe6c89e1eef0cf82da1592a6b2: Status 404 returned error can't find the container with id 56564c56aa716d3c3d38a8e6ad5553476643b7fe6c89e1eef0cf82da1592a6b2 Nov 03 15:08:25.174253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:25.174224 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6d2mt"] Nov 03 15:08:25.177061 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:25.177035 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c47852_6ab2_4c63_bb3f_ec5a3d7633c6.slice/crio-ad83b69fe6ca132caeec378dd533e0e58f69ef1510b1d3315753d95279c737ab WatchSource:0}: Error finding container ad83b69fe6ca132caeec378dd533e0e58f69ef1510b1d3315753d95279c737ab: Status 404 returned error can't find the container with id ad83b69fe6ca132caeec378dd533e0e58f69ef1510b1d3315753d95279c737ab Nov 03 15:08:25.225801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:25.225745 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-2drvj"] Nov 03 15:08:25.229080 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:25.229057 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d582038_d053_4b49_b302_0b7188666463.slice/crio-fededc07c4d7ba3531bc938ec520d11aea672a97dca0c06392ad39d694827336 WatchSource:0}: Error finding container fededc07c4d7ba3531bc938ec520d11aea672a97dca0c06392ad39d694827336: Status 404 returned error can't find the container with id fededc07c4d7ba3531bc938ec520d11aea672a97dca0c06392ad39d694827336 Nov 03 15:08:25.248935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:25.248904 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p8qkv"] Nov 03 15:08:25.251092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:25.251073 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6fvtf"] Nov 03 15:08:25.252494 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:25.252453 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84eeee2e_89d6_4cf1_aa44_76ebb65a0dd6.slice/crio-253d407a99bd221a9dd1c2986d67c536105d1a1ce4eaf6a23188b34a1e933ed9 WatchSource:0}: Error finding container 253d407a99bd221a9dd1c2986d67c536105d1a1ce4eaf6a23188b34a1e933ed9: Status 404 returned error can't find the container with id 253d407a99bd221a9dd1c2986d67c536105d1a1ce4eaf6a23188b34a1e933ed9 Nov 03 15:08:25.254187 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:25.254161 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1381acc6_37dc_484b_be29_9f9c617af63d.slice/crio-ac72d6839fc5e2daf27c31de1b2fb3915499345358aefe50bb656961f13691ad WatchSource:0}: Error finding container ac72d6839fc5e2daf27c31de1b2fb3915499345358aefe50bb656961f13691ad: Status 404 returned error can't find the container with id ac72d6839fc5e2daf27c31de1b2fb3915499345358aefe50bb656961f13691ad Nov 03 15:08:25.735973 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:25.735937 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-2drvj" event={"ID":"6d582038-d053-4b49-b302-0b7188666463","Type":"ContainerStarted","Data":"fededc07c4d7ba3531bc938ec520d11aea672a97dca0c06392ad39d694827336"} Nov 03 15:08:25.737144 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:25.737117 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kfw9b" event={"ID":"9846b382-908f-452b-afb6-083dfc331d62","Type":"ContainerStarted","Data":"56564c56aa716d3c3d38a8e6ad5553476643b7fe6c89e1eef0cf82da1592a6b2"} Nov 03 15:08:25.738506 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:25.738482 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p8qkv" event={"ID":"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6","Type":"ContainerStarted","Data":"03a01a07b061faac8b83da8dabc3a32ee4c16b7a70e89b9c9f099418353c55fc"} Nov 03 15:08:25.738599 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:25.738527 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p8qkv" event={"ID":"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6","Type":"ContainerStarted","Data":"253d407a99bd221a9dd1c2986d67c536105d1a1ce4eaf6a23188b34a1e933ed9"} Nov 03 15:08:25.739528 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:25.739512 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6fvtf" event={"ID":"1381acc6-37dc-484b-be29-9f9c617af63d","Type":"ContainerStarted","Data":"ac72d6839fc5e2daf27c31de1b2fb3915499345358aefe50bb656961f13691ad"} Nov 03 15:08:25.740519 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:25.740499 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6d2mt" event={"ID":"c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6","Type":"ContainerStarted","Data":"ad83b69fe6ca132caeec378dd533e0e58f69ef1510b1d3315753d95279c737ab"} Nov 03 15:08:27.454313 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.454250 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-77965c4cf-qdrhq"] Nov 03 15:08:27.456427 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.456386 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.509777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.509743 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 15:08:27.509901 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.509832 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 15:08:27.537878 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.537813 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5976\" (UniqueName: \"kubernetes.io/projected/28bbf520-6347-4c53-992f-2d23875c02b2-kube-api-access-b5976\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.537878 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.537864 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-oauth-serving-cert\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.538066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.537904 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28bbf520-6347-4c53-992f-2d23875c02b2-console-serving-cert\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.538066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.537927 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28bbf520-6347-4c53-992f-2d23875c02b2-console-oauth-config\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.538066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.537955 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-service-ca\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.538066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.537982 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-console-config\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.538507 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.538475 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28bbf520-6347-4c53-992f-2d23875c02b2-tmp\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.558976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.558951 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 15:08:27.559100 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.559053 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mdmnc\"" Nov 03 15:08:27.559100 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.559063 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 15:08:27.559262 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.559240 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 15:08:27.559996 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.559952 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77965c4cf-qdrhq"] Nov 03 15:08:27.638961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.638929 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5976\" (UniqueName: \"kubernetes.io/projected/28bbf520-6347-4c53-992f-2d23875c02b2-kube-api-access-b5976\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.639172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.638985 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-oauth-serving-cert\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.639172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.639025 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28bbf520-6347-4c53-992f-2d23875c02b2-console-serving-cert\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.639172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.639048 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28bbf520-6347-4c53-992f-2d23875c02b2-console-oauth-config\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.639172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.639075 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-service-ca\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.639172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.639100 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-console-config\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.639172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.639133 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28bbf520-6347-4c53-992f-2d23875c02b2-tmp\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.641658 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.639460 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28bbf520-6347-4c53-992f-2d23875c02b2-tmp\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.641658 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.640427 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-service-ca\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.641658 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.641033 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-oauth-serving-cert\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.641658 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.641570 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-console-config\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.657222 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.657191 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28bbf520-6347-4c53-992f-2d23875c02b2-console-oauth-config\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.657344 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.657258 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28bbf520-6347-4c53-992f-2d23875c02b2-console-serving-cert\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.702352 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.702302 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5976\" (UniqueName: \"kubernetes.io/projected/28bbf520-6347-4c53-992f-2d23875c02b2-kube-api-access-b5976\") pod \"console-77965c4cf-qdrhq\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.755748 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.755007 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kfw9b" event={"ID":"9846b382-908f-452b-afb6-083dfc331d62","Type":"ContainerStarted","Data":"4f6bb5a47a56ed93d9de72f459f265d6e29b8fb7500883c94616390e8f0c2522"} Nov 03 15:08:27.757622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.757596 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kfw9b" Nov 03 15:08:27.760385 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.760359 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p8qkv" event={"ID":"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6","Type":"ContainerStarted","Data":"304b6fbaa51c24de93d266222446630df051f74a882d0b83b85147e809fc157b"} Nov 03 15:08:27.762483 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.762433 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kfw9b" Nov 03 15:08:27.770035 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.770013 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:27.872512 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:27.872461 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kfw9b" podStartSLOduration=1.665877882 podStartE2EDuration="3.872443047s" totalImagesPullingTime="2.206565165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:27.871773513 +0000 UTC m=+15.302731317" watchObservedRunningTime="2026-11-03 15:08:27.872443047 +0000 UTC m=+15.303400854" Nov 03 15:08:28.208296 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:28.208248 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77965c4cf-qdrhq"] Nov 03 15:08:28.212951 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:28.212907 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28bbf520_6347_4c53_992f_2d23875c02b2.slice/crio-f9de843d0e75874ea764beb56e94261f6173445cae95beec3ee462520f2301d3 WatchSource:0}: Error finding container f9de843d0e75874ea764beb56e94261f6173445cae95beec3ee462520f2301d3: Status 404 returned error can't find the container with id f9de843d0e75874ea764beb56e94261f6173445cae95beec3ee462520f2301d3 Nov 03 15:08:28.766887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:28.766738 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6d2mt" event={"ID":"c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6","Type":"ContainerStarted","Data":"b0da9315ff322766571dd13b649f697024d84a08f10718c6ea96a92ed0132427"} Nov 03 15:08:28.766887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:28.766797 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6d2mt" event={"ID":"c3c47852-6ab2-4c63-bb3f-ec5a3d7633c6","Type":"ContainerStarted","Data":"ff1446c501b4758331bd7b22231515aaa5a4aec1ba347553f829b975ceb1e09e"} Nov 03 15:08:28.767377 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:28.766890 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:28.769344 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:28.769293 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77965c4cf-qdrhq" event={"ID":"28bbf520-6347-4c53-992f-2d23875c02b2","Type":"ContainerStarted","Data":"f9de843d0e75874ea764beb56e94261f6173445cae95beec3ee462520f2301d3"} Nov 03 15:08:28.889501 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:28.889439 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-6d2mt" podStartSLOduration=1.739045546 podStartE2EDuration="4.889419006s" totalImagesPullingTime="3.15037346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:28.888399734 +0000 UTC m=+16.319357542" watchObservedRunningTime="2026-11-03 15:08:28.889419006 +0000 UTC m=+16.320376813" Nov 03 15:08:28.950109 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:28.950072 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-pqswq"] Nov 03 15:08:28.952349 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:28.952325 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" Nov 03 15:08:29.005203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.005174 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 15:08:29.005368 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.005338 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 15:08:29.005606 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.005587 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 15:08:29.005672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.005653 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ntgct\"" Nov 03 15:08:29.005778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.005763 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 15:08:29.005857 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.005838 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 15:08:29.012669 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.012599 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-pqswq"] Nov 03 15:08:29.048808 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.048727 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/41e3b9c7-b796-4aab-8324-4c7a0a6c9429-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-pqswq\" (UID: \"41e3b9c7-b796-4aab-8324-4c7a0a6c9429\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" Nov 03 15:08:29.048955 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.048807 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41e3b9c7-b796-4aab-8324-4c7a0a6c9429-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-pqswq\" (UID: \"41e3b9c7-b796-4aab-8324-4c7a0a6c9429\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" Nov 03 15:08:29.048955 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.048838 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41e3b9c7-b796-4aab-8324-4c7a0a6c9429-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-pqswq\" (UID: \"41e3b9c7-b796-4aab-8324-4c7a0a6c9429\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" Nov 03 15:08:29.048955 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.048863 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw2gs\" (UniqueName: \"kubernetes.io/projected/41e3b9c7-b796-4aab-8324-4c7a0a6c9429-kube-api-access-nw2gs\") pod \"prometheus-operator-9cd89b74c-pqswq\" (UID: \"41e3b9c7-b796-4aab-8324-4c7a0a6c9429\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" Nov 03 15:08:29.149665 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.149626 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41e3b9c7-b796-4aab-8324-4c7a0a6c9429-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-pqswq\" (UID: \"41e3b9c7-b796-4aab-8324-4c7a0a6c9429\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" Nov 03 15:08:29.149665 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.149667 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41e3b9c7-b796-4aab-8324-4c7a0a6c9429-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-pqswq\" (UID: \"41e3b9c7-b796-4aab-8324-4c7a0a6c9429\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" Nov 03 15:08:29.149934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.149718 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw2gs\" (UniqueName: \"kubernetes.io/projected/41e3b9c7-b796-4aab-8324-4c7a0a6c9429-kube-api-access-nw2gs\") pod \"prometheus-operator-9cd89b74c-pqswq\" (UID: \"41e3b9c7-b796-4aab-8324-4c7a0a6c9429\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" Nov 03 15:08:29.149934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.149753 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/41e3b9c7-b796-4aab-8324-4c7a0a6c9429-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-pqswq\" (UID: \"41e3b9c7-b796-4aab-8324-4c7a0a6c9429\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" Nov 03 15:08:29.151627 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.151580 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41e3b9c7-b796-4aab-8324-4c7a0a6c9429-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-pqswq\" (UID: \"41e3b9c7-b796-4aab-8324-4c7a0a6c9429\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" Nov 03 15:08:29.153516 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.153489 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/41e3b9c7-b796-4aab-8324-4c7a0a6c9429-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-pqswq\" (UID: \"41e3b9c7-b796-4aab-8324-4c7a0a6c9429\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" Nov 03 15:08:29.153609 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.153591 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41e3b9c7-b796-4aab-8324-4c7a0a6c9429-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-pqswq\" (UID: \"41e3b9c7-b796-4aab-8324-4c7a0a6c9429\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" Nov 03 15:08:29.214043 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.214010 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw2gs\" (UniqueName: \"kubernetes.io/projected/41e3b9c7-b796-4aab-8324-4c7a0a6c9429-kube-api-access-nw2gs\") pod \"prometheus-operator-9cd89b74c-pqswq\" (UID: \"41e3b9c7-b796-4aab-8324-4c7a0a6c9429\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" Nov 03 15:08:29.265597 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.265575 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" Nov 03 15:08:29.673115 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.673033 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-pqswq"] Nov 03 15:08:29.677020 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:29.676991 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e3b9c7_b796_4aab_8324_4c7a0a6c9429.slice/crio-30c769ebf0d36fb9388576ffc3c66873a16ad8902fdf7d941c507182a2453e50 WatchSource:0}: Error finding container 30c769ebf0d36fb9388576ffc3c66873a16ad8902fdf7d941c507182a2453e50: Status 404 returned error can't find the container with id 30c769ebf0d36fb9388576ffc3c66873a16ad8902fdf7d941c507182a2453e50 Nov 03 15:08:29.781390 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.781314 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p8qkv" event={"ID":"84eeee2e-89d6-4cf1-aa44-76ebb65a0dd6","Type":"ContainerStarted","Data":"b25cb8c727f5dcf60012769867418c73c4700ba5fd783570ff7be4bfaa569c63"} Nov 03 15:08:29.783405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.783376 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6fvtf" event={"ID":"1381acc6-37dc-484b-be29-9f9c617af63d","Type":"ContainerStarted","Data":"6cbe1579a005b5eb1047b3060217a49cc64fe4aa20219bab53802cf4c6050427"} Nov 03 15:08:29.785010 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.784950 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" event={"ID":"41e3b9c7-b796-4aab-8324-4c7a0a6c9429","Type":"ContainerStarted","Data":"30c769ebf0d36fb9388576ffc3c66873a16ad8902fdf7d941c507182a2453e50"} Nov 03 15:08:29.901425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:29.901377 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p8qkv" podStartSLOduration=1.8623410759999999 podStartE2EDuration="5.901363624s" totalImagesPullingTime="4.039022548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:08:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:29.900610339 +0000 UTC m=+17.331568147" watchObservedRunningTime="2026-11-03 15:08:29.901363624 +0000 UTC m=+17.332321430" Nov 03 15:08:30.025509 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:30.025456 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6fvtf" podStartSLOduration=2.019262964 podStartE2EDuration="6.025441738s" totalImagesPullingTime="4.006178774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:30.024010928 +0000 UTC m=+17.454968735" watchObservedRunningTime="2026-11-03 15:08:30.025441738 +0000 UTC m=+17.456399535" Nov 03 15:08:31.796179 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:31.796098 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" event={"ID":"41e3b9c7-b796-4aab-8324-4c7a0a6c9429","Type":"ContainerStarted","Data":"d173d935bfea0bc17ee28e83b5ec22e9c83ce222aad70e20861de35049d95892"} Nov 03 15:08:31.796179 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:31.796172 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" event={"ID":"41e3b9c7-b796-4aab-8324-4c7a0a6c9429","Type":"ContainerStarted","Data":"deadbc3d7c3b02052915be5709f0b6b2b4567ada992be1209c12bbff97d57e53"} Nov 03 15:08:32.801294 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:32.801194 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77965c4cf-qdrhq" event={"ID":"28bbf520-6347-4c53-992f-2d23875c02b2","Type":"ContainerStarted","Data":"a23c86277b4125178fe45b32101b10577f51284a9d98264b54df70f6b0ce6a48"} Nov 03 15:08:32.921078 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:32.921017 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-pqswq" podStartSLOduration=3.123831914 podStartE2EDuration="4.921000615s" totalImagesPullingTime="1.797168701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:31.934899676 +0000 UTC m=+19.365857483" watchObservedRunningTime="2026-11-03 15:08:32.921000615 +0000 UTC m=+20.351958422" Nov 03 15:08:32.921507 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:32.921463 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-77965c4cf-qdrhq" podStartSLOduration=2.180539382 podStartE2EDuration="5.921450378s" totalImagesPullingTime="3.740910996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:32.920087378 +0000 UTC m=+20.351045185" watchObservedRunningTime="2026-11-03 15:08:32.921450378 +0000 UTC m=+20.352408186" Nov 03 15:08:33.996784 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:33.996750 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q5rxt" Nov 03 15:08:34.481871 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.481833 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h"] Nov 03 15:08:34.484728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.484677 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" Nov 03 15:08:34.537497 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.537441 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 15:08:34.587232 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.587167 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 15:08:34.587375 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.587287 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fdw75\"" Nov 03 15:08:34.588360 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.588336 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h"] Nov 03 15:08:34.599657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.599631 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vwqg\" (UniqueName: \"kubernetes.io/projected/dda38876-5e32-4413-bdfc-63feb921f660-kube-api-access-9vwqg\") pod \"openshift-state-metrics-7bd7489957-6vg8h\" (UID: \"dda38876-5e32-4413-bdfc-63feb921f660\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" Nov 03 15:08:34.599786 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.599676 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dda38876-5e32-4413-bdfc-63feb921f660-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6vg8h\" (UID: \"dda38876-5e32-4413-bdfc-63feb921f660\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" Nov 03 15:08:34.599849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.599806 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dda38876-5e32-4413-bdfc-63feb921f660-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6vg8h\" (UID: \"dda38876-5e32-4413-bdfc-63feb921f660\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" Nov 03 15:08:34.599903 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.599864 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dda38876-5e32-4413-bdfc-63feb921f660-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6vg8h\" (UID: \"dda38876-5e32-4413-bdfc-63feb921f660\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" Nov 03 15:08:34.638647 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.638619 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9zb9h"] Nov 03 15:08:34.640940 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.640915 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.642892 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.642871 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224"] Nov 03 15:08:34.645111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.645091 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.696273 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.696246 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 15:08:34.696463 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.696447 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 15:08:34.696557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.696542 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kx5hh\"" Nov 03 15:08:34.697470 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.697446 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 15:08:34.697962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.697931 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 15:08:34.698066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.698019 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zrkw4\"" Nov 03 15:08:34.700306 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.700288 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7f77efd-9feb-4054-848b-fc10fe76a318-sys\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.700407 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.700314 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7f77efd-9feb-4054-848b-fc10fe76a318-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.700407 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.700336 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhc8k\" (UniqueName: \"kubernetes.io/projected/c7f77efd-9feb-4054-848b-fc10fe76a318-kube-api-access-dhc8k\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.700407 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.700352 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.700548 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.700433 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dda38876-5e32-4413-bdfc-63feb921f660-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6vg8h\" (UID: \"dda38876-5e32-4413-bdfc-63feb921f660\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" Nov 03 15:08:34.700548 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.700471 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c7f77efd-9feb-4054-848b-fc10fe76a318-node-exporter-wtmp\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.700885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.700864 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c7f77efd-9feb-4054-848b-fc10fe76a318-node-exporter-textfile\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.700980 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.700954 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7f77efd-9feb-4054-848b-fc10fe76a318-metrics-client-ca\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.701085 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.701010 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dda38876-5e32-4413-bdfc-63feb921f660-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6vg8h\" (UID: \"dda38876-5e32-4413-bdfc-63feb921f660\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" Nov 03 15:08:34.701085 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.701052 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.701192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.701118 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.701192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.701174 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.701288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.701224 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.701288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.701244 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c7f77efd-9feb-4054-848b-fc10fe76a318-root\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.701382 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.701323 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7f77efd-9feb-4054-848b-fc10fe76a318-node-exporter-tls\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.701382 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.701344 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vwqg\" (UniqueName: \"kubernetes.io/projected/dda38876-5e32-4413-bdfc-63feb921f660-kube-api-access-9vwqg\") pod \"openshift-state-metrics-7bd7489957-6vg8h\" (UID: \"dda38876-5e32-4413-bdfc-63feb921f660\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" Nov 03 15:08:34.701467 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.701384 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dda38876-5e32-4413-bdfc-63feb921f660-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6vg8h\" (UID: \"dda38876-5e32-4413-bdfc-63feb921f660\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" Nov 03 15:08:34.701467 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.701415 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c7f77efd-9feb-4054-848b-fc10fe76a318-node-exporter-accelerators-collector-config\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.701467 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.701450 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hmsr\" (UniqueName: \"kubernetes.io/projected/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-kube-api-access-4hmsr\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.702231 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.702214 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dda38876-5e32-4413-bdfc-63feb921f660-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6vg8h\" (UID: \"dda38876-5e32-4413-bdfc-63feb921f660\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" Nov 03 15:08:34.703471 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.703447 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dda38876-5e32-4413-bdfc-63feb921f660-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6vg8h\" (UID: \"dda38876-5e32-4413-bdfc-63feb921f660\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" Nov 03 15:08:34.703614 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.703591 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dda38876-5e32-4413-bdfc-63feb921f660-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6vg8h\" (UID: \"dda38876-5e32-4413-bdfc-63feb921f660\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" Nov 03 15:08:34.703858 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.703833 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 15:08:34.703957 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.703940 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 15:08:34.728175 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.728154 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224"] Nov 03 15:08:34.771939 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.771882 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vwqg\" (UniqueName: \"kubernetes.io/projected/dda38876-5e32-4413-bdfc-63feb921f660-kube-api-access-9vwqg\") pod \"openshift-state-metrics-7bd7489957-6vg8h\" (UID: \"dda38876-5e32-4413-bdfc-63feb921f660\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" Nov 03 15:08:34.798892 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.798864 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" Nov 03 15:08:34.801820 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.801798 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7f77efd-9feb-4054-848b-fc10fe76a318-node-exporter-tls\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.801937 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.801848 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c7f77efd-9feb-4054-848b-fc10fe76a318-node-exporter-accelerators-collector-config\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.801937 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.801880 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hmsr\" (UniqueName: \"kubernetes.io/projected/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-kube-api-access-4hmsr\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.801937 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.801908 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7f77efd-9feb-4054-848b-fc10fe76a318-sys\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.801937 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.801932 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7f77efd-9feb-4054-848b-fc10fe76a318-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.802133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.801955 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhc8k\" (UniqueName: \"kubernetes.io/projected/c7f77efd-9feb-4054-848b-fc10fe76a318-kube-api-access-dhc8k\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.802133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.801980 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.802133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.802003 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7f77efd-9feb-4054-848b-fc10fe76a318-sys\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.802133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.802023 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c7f77efd-9feb-4054-848b-fc10fe76a318-node-exporter-wtmp\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.802133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.802053 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c7f77efd-9feb-4054-848b-fc10fe76a318-node-exporter-textfile\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.802133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.802077 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7f77efd-9feb-4054-848b-fc10fe76a318-metrics-client-ca\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.802133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.802118 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.802454 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.802142 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.802454 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.802173 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.802454 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.802210 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.802454 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.802236 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c7f77efd-9feb-4054-848b-fc10fe76a318-root\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.802454 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.802313 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c7f77efd-9feb-4054-848b-fc10fe76a318-root\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.802454 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.802447 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c7f77efd-9feb-4054-848b-fc10fe76a318-node-exporter-accelerators-collector-config\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.802772 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.802539 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c7f77efd-9feb-4054-848b-fc10fe76a318-node-exporter-wtmp\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.802772 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.802629 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c7f77efd-9feb-4054-848b-fc10fe76a318-node-exporter-textfile\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.802881 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.802786 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7f77efd-9feb-4054-848b-fc10fe76a318-metrics-client-ca\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.802881 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.802863 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.803089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.803070 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.803233 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.803182 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.804756 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.804728 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7f77efd-9feb-4054-848b-fc10fe76a318-node-exporter-tls\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.805359 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.805315 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.805642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.805616 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.805742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.805665 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7f77efd-9feb-4054-848b-fc10fe76a318-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.864409 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.864381 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hmsr\" (UniqueName: \"kubernetes.io/projected/85b5f3bf-df66-4584-8a1b-92610ae8d8dc-kube-api-access-4hmsr\") pod \"kube-state-metrics-5dc8d7d89b-hf224\" (UID: \"85b5f3bf-df66-4584-8a1b-92610ae8d8dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.865409 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.865350 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhc8k\" (UniqueName: \"kubernetes.io/projected/c7f77efd-9feb-4054-848b-fc10fe76a318-kube-api-access-dhc8k\") pod \"node-exporter-9zb9h\" (UID: \"c7f77efd-9feb-4054-848b-fc10fe76a318\") " pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.957531 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.957492 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9zb9h" Nov 03 15:08:34.963382 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:34.963359 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" Nov 03 15:08:34.971735 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:34.971677 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f77efd_9feb_4054_848b_fc10fe76a318.slice/crio-75ca8d9e2fcd7c05ac8c09dc2e0366b1bd6a9f9caa10d97010508f846bfbd8bc WatchSource:0}: Error finding container 75ca8d9e2fcd7c05ac8c09dc2e0366b1bd6a9f9caa10d97010508f846bfbd8bc: Status 404 returned error can't find the container with id 75ca8d9e2fcd7c05ac8c09dc2e0366b1bd6a9f9caa10d97010508f846bfbd8bc Nov 03 15:08:35.177515 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:35.177484 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h"] Nov 03 15:08:35.183869 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:35.183838 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda38876_5e32_4413_bdfc_63feb921f660.slice/crio-f6337a4f6b0a46cac124bfa98865c86676f074e8aa58fc33b3a0ce0116f28107 WatchSource:0}: Error finding container f6337a4f6b0a46cac124bfa98865c86676f074e8aa58fc33b3a0ce0116f28107: Status 404 returned error can't find the container with id f6337a4f6b0a46cac124bfa98865c86676f074e8aa58fc33b3a0ce0116f28107 Nov 03 15:08:35.348816 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:35.347789 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224"] Nov 03 15:08:35.352661 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:35.352635 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b5f3bf_df66_4584_8a1b_92610ae8d8dc.slice/crio-e4c91215ffa68f2830a377095c83c050db422c600cf0802c9a506655d35da6fc WatchSource:0}: Error finding container e4c91215ffa68f2830a377095c83c050db422c600cf0802c9a506655d35da6fc: Status 404 returned error can't find the container with id e4c91215ffa68f2830a377095c83c050db422c600cf0802c9a506655d35da6fc Nov 03 15:08:35.812948 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:35.812918 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" event={"ID":"85b5f3bf-df66-4584-8a1b-92610ae8d8dc","Type":"ContainerStarted","Data":"e4c91215ffa68f2830a377095c83c050db422c600cf0802c9a506655d35da6fc"} Nov 03 15:08:35.814062 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:35.814041 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9zb9h" event={"ID":"c7f77efd-9feb-4054-848b-fc10fe76a318","Type":"ContainerStarted","Data":"75ca8d9e2fcd7c05ac8c09dc2e0366b1bd6a9f9caa10d97010508f846bfbd8bc"} Nov 03 15:08:35.815690 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:35.815671 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" event={"ID":"dda38876-5e32-4413-bdfc-63feb921f660","Type":"ContainerStarted","Data":"118e345441ad3ca683234e943b9f913e992a321cea22e9b0420ec49f1e070d0d"} Nov 03 15:08:35.815777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:35.815728 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" event={"ID":"dda38876-5e32-4413-bdfc-63feb921f660","Type":"ContainerStarted","Data":"fa6a7a96c1a409e94fa3456b851e662d3b9ad775923636de62049a9b10514924"} Nov 03 15:08:35.815777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:35.815747 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" event={"ID":"dda38876-5e32-4413-bdfc-63feb921f660","Type":"ContainerStarted","Data":"f6337a4f6b0a46cac124bfa98865c86676f074e8aa58fc33b3a0ce0116f28107"} Nov 03 15:08:36.824827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:36.824750 4290 generic.go:356] "Generic (PLEG): container finished" podID="c7f77efd-9feb-4054-848b-fc10fe76a318" containerID="c3c2a38dab3b81e4524b84ec65765435b79d93cc01045b5a9ed26179372dc488" exitCode=0 Nov 03 15:08:36.824827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:36.824812 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9zb9h" event={"ID":"c7f77efd-9feb-4054-848b-fc10fe76a318","Type":"ContainerDied","Data":"c3c2a38dab3b81e4524b84ec65765435b79d93cc01045b5a9ed26179372dc488"} Nov 03 15:08:36.941713 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:36.941674 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 15:08:36.950026 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:36.949994 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.016350 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.016322 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 15:08:37.024486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.024391 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfdj6\" (UniqueName: \"kubernetes.io/projected/a5c76303-7f6d-467e-8e89-0d1a925afd02-kube-api-access-kfdj6\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.024486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.024428 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5c76303-7f6d-467e-8e89-0d1a925afd02-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.024486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.024455 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-web-config\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.024693 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.024571 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.024693 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.024608 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a5c76303-7f6d-467e-8e89-0d1a925afd02-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.024693 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.024624 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5c76303-7f6d-467e-8e89-0d1a925afd02-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.024693 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.024671 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.024884 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.024763 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5c76303-7f6d-467e-8e89-0d1a925afd02-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.024884 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.024828 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.024979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.024883 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5c76303-7f6d-467e-8e89-0d1a925afd02-config-out\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.024979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.024907 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.024979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.024923 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.024979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.024949 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-config-volume\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.068550 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.068517 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 15:08:37.089774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.089685 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 15:08:37.089774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.089723 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 15:08:37.089774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.089736 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jxtqz\"" Nov 03 15:08:37.089774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.089685 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 15:08:37.089774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.089759 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 15:08:37.089774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.089783 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 15:08:37.090140 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.089813 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 15:08:37.090140 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.089840 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 15:08:37.090140 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.089903 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 15:08:37.125492 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.125461 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfdj6\" (UniqueName: \"kubernetes.io/projected/a5c76303-7f6d-467e-8e89-0d1a925afd02-kube-api-access-kfdj6\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.125617 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.125510 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5c76303-7f6d-467e-8e89-0d1a925afd02-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.125617 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.125537 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-web-config\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.125617 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.125581 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.125617 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.125605 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a5c76303-7f6d-467e-8e89-0d1a925afd02-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.125801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.125631 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5c76303-7f6d-467e-8e89-0d1a925afd02-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.125801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.125687 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.125801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.125729 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5c76303-7f6d-467e-8e89-0d1a925afd02-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.125801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.125755 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.125801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.125784 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5c76303-7f6d-467e-8e89-0d1a925afd02-config-out\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.125995 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.125810 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.125995 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.125834 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.125995 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.125861 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-config-volume\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.126962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.126295 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a5c76303-7f6d-467e-8e89-0d1a925afd02-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.127442 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.127132 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5c76303-7f6d-467e-8e89-0d1a925afd02-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.127672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.127645 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5c76303-7f6d-467e-8e89-0d1a925afd02-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.129443 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.129411 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5c76303-7f6d-467e-8e89-0d1a925afd02-config-out\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.130603 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.130574 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.131302 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.131282 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-config-volume\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.131513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.131481 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.131613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.131592 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-web-config\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.131961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.131919 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.132256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.132236 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.132939 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.132918 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.145310 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.145287 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5c76303-7f6d-467e-8e89-0d1a925afd02-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.188435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.188409 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfdj6\" (UniqueName: \"kubernetes.io/projected/a5c76303-7f6d-467e-8e89-0d1a925afd02-kube-api-access-kfdj6\") pod \"alertmanager-main-0\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.271669 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.271638 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:08:37.402228 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.402205 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7bd85bcf57-n79t2"] Nov 03 15:08:37.408238 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.408209 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.472674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.472642 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bd85bcf57-n79t2"] Nov 03 15:08:37.516337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.516312 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 15:08:37.516498 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.516478 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 15:08:37.516574 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.516543 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 15:08:37.516663 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.516644 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 15:08:37.516950 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.516934 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 15:08:37.517046 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.516946 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-z8jgh\"" Nov 03 15:08:37.517046 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.516953 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-er83q7p82uga7\"" Nov 03 15:08:37.529222 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.529195 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-grpc-tls\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.529308 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.529243 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/653f8239-2130-49de-9dd8-c45f1f161ee8-metrics-client-ca\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.529308 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.529280 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.529384 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.529308 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cg8d\" (UniqueName: \"kubernetes.io/projected/653f8239-2130-49de-9dd8-c45f1f161ee8-kube-api-access-4cg8d\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.529384 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.529338 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-thanos-querier-tls\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.529456 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.529393 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.529456 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.529418 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.529456 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.529450 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/653f8239-2130-49de-9dd8-c45f1f161ee8-tmp\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.529596 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.529483 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.630760 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.630716 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/653f8239-2130-49de-9dd8-c45f1f161ee8-tmp\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.630760 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.630763 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.630972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.630800 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-grpc-tls\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.630972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.630857 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/653f8239-2130-49de-9dd8-c45f1f161ee8-metrics-client-ca\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.630972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.630886 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.630972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.630911 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cg8d\" (UniqueName: \"kubernetes.io/projected/653f8239-2130-49de-9dd8-c45f1f161ee8-kube-api-access-4cg8d\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.630972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.630945 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-thanos-querier-tls\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.631211 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.630992 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.631211 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.631013 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.631306 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.631210 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/653f8239-2130-49de-9dd8-c45f1f161ee8-tmp\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.632385 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.632353 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/653f8239-2130-49de-9dd8-c45f1f161ee8-metrics-client-ca\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.634732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.634415 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.634732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.634674 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-thanos-querier-tls\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.634861 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.634814 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.634861 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.634833 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-grpc-tls\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.634987 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.634960 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.636332 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.636310 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/653f8239-2130-49de-9dd8-c45f1f161ee8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.693954 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.693926 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cg8d\" (UniqueName: \"kubernetes.io/projected/653f8239-2130-49de-9dd8-c45f1f161ee8-kube-api-access-4cg8d\") pod \"thanos-querier-7bd85bcf57-n79t2\" (UID: \"653f8239-2130-49de-9dd8-c45f1f161ee8\") " pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.724381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.724354 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 15:08:37.727043 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:37.727013 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c76303_7f6d_467e_8e89_0d1a925afd02.slice/crio-b81cf052744fdb489cec35d43911c7b1be4c8ae3749a4257cc3e90ab50d27b1e WatchSource:0}: Error finding container b81cf052744fdb489cec35d43911c7b1be4c8ae3749a4257cc3e90ab50d27b1e: Status 404 returned error can't find the container with id b81cf052744fdb489cec35d43911c7b1be4c8ae3749a4257cc3e90ab50d27b1e Nov 03 15:08:37.739674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.739650 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:37.773722 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.770596 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:37.773722 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.771620 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:37.779720 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.777476 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:37.832136 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.832105 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" event={"ID":"85b5f3bf-df66-4584-8a1b-92610ae8d8dc","Type":"ContainerStarted","Data":"3b3463ebca6008ecec38e3bab78792e2b3ee62dd7b1ef725ebcc96932dad4049"} Nov 03 15:08:37.832508 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.832173 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" event={"ID":"85b5f3bf-df66-4584-8a1b-92610ae8d8dc","Type":"ContainerStarted","Data":"d3fa700db25b10d9e2dd2097a1ed55eaa60b4f6560f1bb44ff83dbdc0c807dfa"} Nov 03 15:08:37.832508 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.832204 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" event={"ID":"85b5f3bf-df66-4584-8a1b-92610ae8d8dc","Type":"ContainerStarted","Data":"ac5ed5883d2ca8ec551f06ed9e8e8f100e233b430b88ef92ded308ba50856a83"} Nov 03 15:08:37.834472 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.834449 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9zb9h" event={"ID":"c7f77efd-9feb-4054-848b-fc10fe76a318","Type":"ContainerStarted","Data":"8ab3f0efcb7f0d1ce5e40b98ec8d588b89084641e5887582d1db54cd9bce837e"} Nov 03 15:08:37.834594 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.834491 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9zb9h" event={"ID":"c7f77efd-9feb-4054-848b-fc10fe76a318","Type":"ContainerStarted","Data":"5ead694234c24f0c4bdcb7871ba8a599ff1680c982e2a47cec7d99465042cffe"} Nov 03 15:08:37.836311 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.836284 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5c76303-7f6d-467e-8e89-0d1a925afd02","Type":"ContainerStarted","Data":"b81cf052744fdb489cec35d43911c7b1be4c8ae3749a4257cc3e90ab50d27b1e"} Nov 03 15:08:37.838453 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.838424 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" event={"ID":"dda38876-5e32-4413-bdfc-63feb921f660","Type":"ContainerStarted","Data":"f8f18cb3588170b552c73f122bafcf0319b76b240a662d70334f6f9ea940fb4f"} Nov 03 15:08:37.842860 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:37.842830 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:08:38.128248 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:38.128186 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hf224" podStartSLOduration=2.171906542 podStartE2EDuration="4.128168116s" totalImagesPullingTime="1.956261574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:38.026660119 +0000 UTC m=+25.457617924" watchObservedRunningTime="2026-11-03 15:08:38.128168116 +0000 UTC m=+25.559125923" Nov 03 15:08:38.129415 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:38.129314 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bd85bcf57-n79t2"] Nov 03 15:08:38.132322 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:38.132293 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653f8239_2130_49de_9dd8_c45f1f161ee8.slice/crio-0e9bd2eab676059139233a5c0423661f14b1aac16fd9dfed58bb712e91e11744 WatchSource:0}: Error finding container 0e9bd2eab676059139233a5c0423661f14b1aac16fd9dfed58bb712e91e11744: Status 404 returned error can't find the container with id 0e9bd2eab676059139233a5c0423661f14b1aac16fd9dfed58bb712e91e11744 Nov 03 15:08:38.264100 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:38.264042 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6vg8h" podStartSLOduration=2.712923279 podStartE2EDuration="4.264023816s" totalImagesPullingTime="1.551100537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:38.263438405 +0000 UTC m=+25.694396212" watchObservedRunningTime="2026-11-03 15:08:38.264023816 +0000 UTC m=+25.694981623" Nov 03 15:08:38.385329 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:38.385222 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9zb9h" podStartSLOduration=2.766219831 podStartE2EDuration="4.385202154s" totalImagesPullingTime="1.52670844s" totalInitContainerRuntime="92.273883ms" isStatefulPod=true podCreationTimestamp="2026-11-03 15:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:38.384080718 +0000 UTC m=+25.815038525" watchObservedRunningTime="2026-11-03 15:08:38.385202154 +0000 UTC m=+25.816159960" Nov 03 15:08:38.705678 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:08:38.705592 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:08:38.788145 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:38.788044 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6d2mt" Nov 03 15:08:38.845021 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:38.844974 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" event={"ID":"653f8239-2130-49de-9dd8-c45f1f161ee8","Type":"ContainerStarted","Data":"0e9bd2eab676059139233a5c0423661f14b1aac16fd9dfed58bb712e91e11744"} Nov 03 15:08:39.385527 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.385498 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-8smb5"] Nov 03 15:08:39.388992 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.388973 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8smb5" Nov 03 15:08:39.446138 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.446114 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-8smb5"] Nov 03 15:08:39.448325 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.448297 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkklg\" (UniqueName: \"kubernetes.io/projected/3ca19e3b-71f2-4067-a8ae-1a4b5afe14dd-kube-api-access-kkklg\") pod \"monitoring-plugin-77f4f76594-8smb5\" (UID: \"3ca19e3b-71f2-4067-a8ae-1a4b5afe14dd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8smb5" Nov 03 15:08:39.448434 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.448349 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca19e3b-71f2-4067-a8ae-1a4b5afe14dd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-8smb5\" (UID: \"3ca19e3b-71f2-4067-a8ae-1a4b5afe14dd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8smb5" Nov 03 15:08:39.503558 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.503521 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nc6gj\"" Nov 03 15:08:39.503742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.503675 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 15:08:39.549849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.549824 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkklg\" (UniqueName: \"kubernetes.io/projected/3ca19e3b-71f2-4067-a8ae-1a4b5afe14dd-kube-api-access-kkklg\") pod \"monitoring-plugin-77f4f76594-8smb5\" (UID: \"3ca19e3b-71f2-4067-a8ae-1a4b5afe14dd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8smb5" Nov 03 15:08:39.550043 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.549862 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca19e3b-71f2-4067-a8ae-1a4b5afe14dd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-8smb5\" (UID: \"3ca19e3b-71f2-4067-a8ae-1a4b5afe14dd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8smb5" Nov 03 15:08:39.553007 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.552983 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca19e3b-71f2-4067-a8ae-1a4b5afe14dd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-8smb5\" (UID: \"3ca19e3b-71f2-4067-a8ae-1a4b5afe14dd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8smb5" Nov 03 15:08:39.617373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.617313 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkklg\" (UniqueName: \"kubernetes.io/projected/3ca19e3b-71f2-4067-a8ae-1a4b5afe14dd-kube-api-access-kkklg\") pod \"monitoring-plugin-77f4f76594-8smb5\" (UID: \"3ca19e3b-71f2-4067-a8ae-1a4b5afe14dd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8smb5" Nov 03 15:08:39.689382 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.689354 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6699cbb5b6-xhr2d"] Nov 03 15:08:39.692147 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.691598 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.702855 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.702821 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8smb5" Nov 03 15:08:39.746559 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.746503 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 15:08:39.746559 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.746504 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hvxht\"" Nov 03 15:08:39.746798 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.746659 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 15:08:39.746798 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.746742 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-djuqeqe3rm0td\"" Nov 03 15:08:39.752436 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.752406 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2d45ed90-f053-4277-85a0-fde14f7ae154-client-ca-bundle\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.752563 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.752455 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d45ed90-f053-4277-85a0-fde14f7ae154-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.752563 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.752477 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8hx4\" (UniqueName: \"kubernetes.io/projected/2d45ed90-f053-4277-85a0-fde14f7ae154-kube-api-access-w8hx4\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.752563 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.752502 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2d45ed90-f053-4277-85a0-fde14f7ae154-audit-log\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.752563 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.752519 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2d45ed90-f053-4277-85a0-fde14f7ae154-secret-metrics-server-tls\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.752563 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.752549 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2d45ed90-f053-4277-85a0-fde14f7ae154-secret-metrics-server-client-certs\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.752563 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.752563 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2d45ed90-f053-4277-85a0-fde14f7ae154-metrics-server-audit-profiles\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.759457 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.759427 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6699cbb5b6-xhr2d"] Nov 03 15:08:39.827603 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.827562 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 15:08:39.827787 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.827665 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 15:08:39.851839 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.851806 4290 generic.go:356] "Generic (PLEG): container finished" podID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerID="1519fb1aa0e1911f44812d9409c6c5b0d2989547766266e99b9ded37302d1afe" exitCode=0 Nov 03 15:08:39.852275 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.851953 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5c76303-7f6d-467e-8e89-0d1a925afd02","Type":"ContainerDied","Data":"1519fb1aa0e1911f44812d9409c6c5b0d2989547766266e99b9ded37302d1afe"} Nov 03 15:08:39.852943 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.852873 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d45ed90-f053-4277-85a0-fde14f7ae154-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.852943 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.852906 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8hx4\" (UniqueName: \"kubernetes.io/projected/2d45ed90-f053-4277-85a0-fde14f7ae154-kube-api-access-w8hx4\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.852943 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.852942 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2d45ed90-f053-4277-85a0-fde14f7ae154-audit-log\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.853124 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.852969 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2d45ed90-f053-4277-85a0-fde14f7ae154-secret-metrics-server-tls\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.853124 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.853013 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2d45ed90-f053-4277-85a0-fde14f7ae154-secret-metrics-server-client-certs\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.853124 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.853038 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2d45ed90-f053-4277-85a0-fde14f7ae154-metrics-server-audit-profiles\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.853124 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.853095 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2d45ed90-f053-4277-85a0-fde14f7ae154-client-ca-bundle\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.853637 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.853572 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d45ed90-f053-4277-85a0-fde14f7ae154-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.854607 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.854585 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2d45ed90-f053-4277-85a0-fde14f7ae154-audit-log\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.855237 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.855212 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2d45ed90-f053-4277-85a0-fde14f7ae154-metrics-server-audit-profiles\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.856811 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.856787 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2d45ed90-f053-4277-85a0-fde14f7ae154-client-ca-bundle\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.857095 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.857078 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2d45ed90-f053-4277-85a0-fde14f7ae154-secret-metrics-server-tls\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.857180 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.857080 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2d45ed90-f053-4277-85a0-fde14f7ae154-secret-metrics-server-client-certs\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:39.917924 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:39.917898 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8hx4\" (UniqueName: \"kubernetes.io/projected/2d45ed90-f053-4277-85a0-fde14f7ae154-kube-api-access-w8hx4\") pod \"metrics-server-6699cbb5b6-xhr2d\" (UID: \"2d45ed90-f053-4277-85a0-fde14f7ae154\") " pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:40.007330 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.007300 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:08:40.114729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.114581 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-8smb5"] Nov 03 15:08:40.117834 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:40.117796 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca19e3b_71f2_4067_a8ae_1a4b5afe14dd.slice/crio-363bf773f36802f205e1c9ae00d4758c2017c465be36cc108d64a025608f0cf5 WatchSource:0}: Error finding container 363bf773f36802f205e1c9ae00d4758c2017c465be36cc108d64a025608f0cf5: Status 404 returned error can't find the container with id 363bf773f36802f205e1c9ae00d4758c2017c465be36cc108d64a025608f0cf5 Nov 03 15:08:40.393769 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.393730 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-f87df669-jfhpm"] Nov 03 15:08:40.397572 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.397549 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6699cbb5b6-xhr2d"] Nov 03 15:08:40.397750 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.397687 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.400829 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:40.400801 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d45ed90_f053_4277_85a0_fde14f7ae154.slice/crio-deb5f20836452e1371922a10ec7f243279bd935d02358f9fe52c2cc7714b29a7 WatchSource:0}: Error finding container deb5f20836452e1371922a10ec7f243279bd935d02358f9fe52c2cc7714b29a7: Status 404 returned error can't find the container with id deb5f20836452e1371922a10ec7f243279bd935d02358f9fe52c2cc7714b29a7 Nov 03 15:08:40.450512 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.450475 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 15:08:40.450512 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.450483 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 15:08:40.450867 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.450834 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 15:08:40.450970 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.450909 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 15:08:40.451042 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.451021 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qwc6g\"" Nov 03 15:08:40.451270 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.451242 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 15:08:40.459676 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.459639 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/442059db-4817-43db-8096-2f9f0511c71c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.459838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.459784 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/442059db-4817-43db-8096-2f9f0511c71c-serving-certs-ca-bundle\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.459838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.459822 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/442059db-4817-43db-8096-2f9f0511c71c-federate-client-tls\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.459953 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.459872 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/442059db-4817-43db-8096-2f9f0511c71c-metrics-client-ca\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.460016 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.459991 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwmrm\" (UniqueName: \"kubernetes.io/projected/442059db-4817-43db-8096-2f9f0511c71c-kube-api-access-qwmrm\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.460127 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.460034 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/442059db-4817-43db-8096-2f9f0511c71c-secret-telemeter-client\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.460127 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.460057 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/442059db-4817-43db-8096-2f9f0511c71c-telemeter-client-tls\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.460127 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.460113 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/442059db-4817-43db-8096-2f9f0511c71c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.501333 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.501295 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 15:08:40.503030 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.502955 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f87df669-jfhpm"] Nov 03 15:08:40.560824 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.560793 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/442059db-4817-43db-8096-2f9f0511c71c-serving-certs-ca-bundle\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.560970 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.560833 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/442059db-4817-43db-8096-2f9f0511c71c-federate-client-tls\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.560970 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.560868 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/442059db-4817-43db-8096-2f9f0511c71c-metrics-client-ca\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.562334 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.561193 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwmrm\" (UniqueName: \"kubernetes.io/projected/442059db-4817-43db-8096-2f9f0511c71c-kube-api-access-qwmrm\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.562334 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.561262 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/442059db-4817-43db-8096-2f9f0511c71c-secret-telemeter-client\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.562334 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.561293 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/442059db-4817-43db-8096-2f9f0511c71c-telemeter-client-tls\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.562334 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.561363 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/442059db-4817-43db-8096-2f9f0511c71c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.562334 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.561395 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/442059db-4817-43db-8096-2f9f0511c71c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.562334 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.562277 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/442059db-4817-43db-8096-2f9f0511c71c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.562334 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.562329 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/442059db-4817-43db-8096-2f9f0511c71c-serving-certs-ca-bundle\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.562777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.562340 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/442059db-4817-43db-8096-2f9f0511c71c-metrics-client-ca\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.564539 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.564512 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/442059db-4817-43db-8096-2f9f0511c71c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.564912 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.564882 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/442059db-4817-43db-8096-2f9f0511c71c-federate-client-tls\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.565074 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.565055 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/442059db-4817-43db-8096-2f9f0511c71c-secret-telemeter-client\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.565188 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.565166 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/442059db-4817-43db-8096-2f9f0511c71c-telemeter-client-tls\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.623750 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.623692 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwmrm\" (UniqueName: \"kubernetes.io/projected/442059db-4817-43db-8096-2f9f0511c71c-kube-api-access-qwmrm\") pod \"telemeter-client-f87df669-jfhpm\" (UID: \"442059db-4817-43db-8096-2f9f0511c71c\") " pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.720139 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.719854 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" Nov 03 15:08:40.860412 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.860341 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" event={"ID":"2d45ed90-f053-4277-85a0-fde14f7ae154","Type":"ContainerStarted","Data":"deb5f20836452e1371922a10ec7f243279bd935d02358f9fe52c2cc7714b29a7"} Nov 03 15:08:40.863120 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:40.863091 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8smb5" event={"ID":"3ca19e3b-71f2-4067-a8ae-1a4b5afe14dd","Type":"ContainerStarted","Data":"363bf773f36802f205e1c9ae00d4758c2017c465be36cc108d64a025608f0cf5"} Nov 03 15:08:41.170506 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:41.170470 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f87df669-jfhpm"] Nov 03 15:08:41.870978 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:41.870884 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" event={"ID":"442059db-4817-43db-8096-2f9f0511c71c","Type":"ContainerStarted","Data":"ddea305975dd6eb74a2ac47072838e9c74632ed517af20755971b24a719637e7"} Nov 03 15:08:41.874854 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:41.874821 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" event={"ID":"653f8239-2130-49de-9dd8-c45f1f161ee8","Type":"ContainerStarted","Data":"73054b7de02dca92dc201cf854e8881a0301caa72ab10c0f16c7e58fce789eb3"} Nov 03 15:08:41.874982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:41.874902 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" event={"ID":"653f8239-2130-49de-9dd8-c45f1f161ee8","Type":"ContainerStarted","Data":"01f8878635f4471d2877f175904eeed1463d07a5f1e61d94bb52d9039b9dbd1c"} Nov 03 15:08:41.874982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:41.874947 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" event={"ID":"653f8239-2130-49de-9dd8-c45f1f161ee8","Type":"ContainerStarted","Data":"85c177fc9aeedea182f1b0c4bf7da9e03b7296de0da8794ef4da3ab0b1dbb69c"} Nov 03 15:08:43.439371 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:43.438531 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qwzv7" Nov 03 15:08:43.888575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:43.888546 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8smb5" event={"ID":"3ca19e3b-71f2-4067-a8ae-1a4b5afe14dd","Type":"ContainerStarted","Data":"3ac5f1da248f0d8ecb2af216027f49ef76366f609e2c1480b1d260999a69e994"} Nov 03 15:08:43.888713 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:43.888606 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8smb5" Nov 03 15:08:43.893408 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:43.893381 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5c76303-7f6d-467e-8e89-0d1a925afd02","Type":"ContainerStarted","Data":"bfbfa0fe467b43cf3f24648a566983ac75751a96767de061fbb082f589c1254b"} Nov 03 15:08:43.893551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:43.893448 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5c76303-7f6d-467e-8e89-0d1a925afd02","Type":"ContainerStarted","Data":"2014885e775cc7842e0578ce1525a278cb9f7bb0e6cc6b6f8ecbb01f46e3a54d"} Nov 03 15:08:43.893551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:43.893476 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5c76303-7f6d-467e-8e89-0d1a925afd02","Type":"ContainerStarted","Data":"773b75d1d36e4e8cee6bd7438648b6228ebae6342047d29c40ec32ac12d07f74"} Nov 03 15:08:43.893551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:43.893506 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5c76303-7f6d-467e-8e89-0d1a925afd02","Type":"ContainerStarted","Data":"142d1ac12130abd4b535299fd06c075e44a81605826754153e1dfaf775f49b57"} Nov 03 15:08:43.893551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:43.893529 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5c76303-7f6d-467e-8e89-0d1a925afd02","Type":"ContainerStarted","Data":"fc07a8baaa6b80d200bb63afe714864fec646bd5a5570fa6d040525662b120ce"} Nov 03 15:08:43.893763 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:43.893555 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5c76303-7f6d-467e-8e89-0d1a925afd02","Type":"ContainerStarted","Data":"cfac535e02b408db6bf6b44bdb81cfa6381b46c9898af2345d25f9cf4d671151"} Nov 03 15:08:43.894786 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:43.894768 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8smb5" Nov 03 15:08:43.897291 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:43.897264 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" event={"ID":"653f8239-2130-49de-9dd8-c45f1f161ee8","Type":"ContainerStarted","Data":"4d69a1c7ab2e75f65769b0db67354b32bfef6224a046a5e5c25478acba08ab72"} Nov 03 15:08:43.897371 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:43.897323 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" event={"ID":"653f8239-2130-49de-9dd8-c45f1f161ee8","Type":"ContainerStarted","Data":"964b3176e402e682d0644ffaaf57275af1b92e6cc15768a860db03fdaec759e4"} Nov 03 15:08:43.899064 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:43.899037 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" event={"ID":"2d45ed90-f053-4277-85a0-fde14f7ae154","Type":"ContainerStarted","Data":"a9a4249a977389bd0b54775603f512be6a27b54ec69a5e382c1a5bc4ea53debf"} Nov 03 15:08:44.011318 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:44.010982 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8smb5" podStartSLOduration=2.1991780419999998 podStartE2EDuration="5.010963856s" totalImagesPullingTime="2.811785814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:44.010957824 +0000 UTC m=+31.441915635" watchObservedRunningTime="2026-11-03 15:08:44.010963856 +0000 UTC m=+31.441921662" Nov 03 15:08:44.280905 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:44.280861 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.583530171 podStartE2EDuration="8.280845444s" totalImagesPullingTime="4.679902489s" totalInitContainerRuntime="17.412784ms" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:44.278170257 +0000 UTC m=+31.709128074" watchObservedRunningTime="2026-11-03 15:08:44.280845444 +0000 UTC m=+31.711803249" Nov 03 15:08:44.907833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:44.907795 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" event={"ID":"442059db-4817-43db-8096-2f9f0511c71c","Type":"ContainerStarted","Data":"866b5636d45f893ce2a142235542b5c6902596adc99ca45d37ac5d503755880c"} Nov 03 15:08:44.908294 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:44.907869 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" event={"ID":"442059db-4817-43db-8096-2f9f0511c71c","Type":"ContainerStarted","Data":"e492d3698f1ccc006ce6879d9faf27827b08e810a5df2454d3d2f14e3c4ebe96"} Nov 03 15:08:44.908294 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:44.907903 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" event={"ID":"442059db-4817-43db-8096-2f9f0511c71c","Type":"ContainerStarted","Data":"cf361c67d118c0f15d5a78ba472fe5fcfc936edc5378f955e9bbd92adb5b3885"} Nov 03 15:08:44.911905 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:44.911857 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" event={"ID":"653f8239-2130-49de-9dd8-c45f1f161ee8","Type":"ContainerStarted","Data":"a7412236c7deafdd77d71be68dd84e99854c03cf340a517701fca6a964db73fd"} Nov 03 15:08:44.912398 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:44.912379 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:45.038122 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:45.038065 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" podStartSLOduration=3.311501435 podStartE2EDuration="6.038043751s" totalImagesPullingTime="2.726542316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:44.403010886 +0000 UTC m=+31.833968692" watchObservedRunningTime="2026-11-03 15:08:45.038043751 +0000 UTC m=+32.469001557" Nov 03 15:08:45.169216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:45.169107 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-f87df669-jfhpm" podStartSLOduration=2.095036073 podStartE2EDuration="5.16908629s" totalImagesPullingTime="3.074050217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:45.039281885 +0000 UTC m=+32.470239690" watchObservedRunningTime="2026-11-03 15:08:45.16908629 +0000 UTC m=+32.600044096" Nov 03 15:08:45.169628 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:45.169596 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" podStartSLOduration=2.898608222 podStartE2EDuration="8.169585193s" totalImagesPullingTime="5.270976971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:45.167192863 +0000 UTC m=+32.598150670" watchObservedRunningTime="2026-11-03 15:08:45.169585193 +0000 UTC m=+32.600543000" Nov 03 15:08:46.925927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:46.925889 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-2drvj" event={"ID":"6d582038-d053-4b49-b302-0b7188666463","Type":"ContainerStarted","Data":"04baf70a52829398b4b6c0aee90b5d2962085a40a419c9530418135dda21effe"} Nov 03 15:08:46.926414 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:46.925958 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-2drvj" Nov 03 15:08:46.948770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:46.948743 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-2drvj" Nov 03 15:08:47.148371 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.148316 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-2drvj" podStartSLOduration=2.472234904 podStartE2EDuration="23.148300991s" totalImagesPullingTime="20.676066087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:47.147033549 +0000 UTC m=+34.577991356" watchObservedRunningTime="2026-11-03 15:08:47.148300991 +0000 UTC m=+34.579258798" Nov 03 15:08:47.150159 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.150133 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-774c544669-f6n62"] Nov 03 15:08:47.180417 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.180349 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.185725 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.184242 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774c544669-f6n62"] Nov 03 15:08:47.310952 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.310897 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 15:08:47.333277 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.333240 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd5eb143-6167-431f-a09c-5a30a311d67e-console-serving-cert\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.333442 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.333370 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd5eb143-6167-431f-a09c-5a30a311d67e-console-oauth-config\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.333442 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.333411 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-service-ca\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.333571 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.333463 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-trusted-ca-bundle\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.333571 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.333524 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-console-config\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.333670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.333579 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-oauth-serving-cert\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.333670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.333611 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f8ct\" (UniqueName: \"kubernetes.io/projected/bd5eb143-6167-431f-a09c-5a30a311d67e-kube-api-access-5f8ct\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.333670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.333637 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd5eb143-6167-431f-a09c-5a30a311d67e-tmp\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.434411 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.434327 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-oauth-serving-cert\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.434411 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.434375 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f8ct\" (UniqueName: \"kubernetes.io/projected/bd5eb143-6167-431f-a09c-5a30a311d67e-kube-api-access-5f8ct\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.434411 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.434404 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd5eb143-6167-431f-a09c-5a30a311d67e-tmp\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.434733 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.434452 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd5eb143-6167-431f-a09c-5a30a311d67e-console-serving-cert\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.434733 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.434525 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd5eb143-6167-431f-a09c-5a30a311d67e-console-oauth-config\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.434733 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.434552 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-service-ca\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.434733 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.434575 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-trusted-ca-bundle\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.434733 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.434604 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-console-config\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.434999 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.434931 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd5eb143-6167-431f-a09c-5a30a311d67e-tmp\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.435304 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.435278 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-oauth-serving-cert\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.435485 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.435307 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-console-config\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.435587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.435538 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-service-ca\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.435841 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.435812 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-trusted-ca-bundle\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.438274 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.438248 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd5eb143-6167-431f-a09c-5a30a311d67e-console-serving-cert\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.438374 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.438317 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd5eb143-6167-431f-a09c-5a30a311d67e-console-oauth-config\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.497223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.497194 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f8ct\" (UniqueName: \"kubernetes.io/projected/bd5eb143-6167-431f-a09c-5a30a311d67e-kube-api-access-5f8ct\") pod \"console-774c544669-f6n62\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.513900 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.513869 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:47.934757 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:47.930534 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774c544669-f6n62"] Nov 03 15:08:47.937249 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:08:47.937212 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5eb143_6167_431f_a09c_5a30a311d67e.slice/crio-7f0acfc84fb5fc598e1bf0246409362df7b196a91e05d6e06333a0e059659cf6 WatchSource:0}: Error finding container 7f0acfc84fb5fc598e1bf0246409362df7b196a91e05d6e06333a0e059659cf6: Status 404 returned error can't find the container with id 7f0acfc84fb5fc598e1bf0246409362df7b196a91e05d6e06333a0e059659cf6 Nov 03 15:08:48.942942 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:48.942907 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774c544669-f6n62" event={"ID":"bd5eb143-6167-431f-a09c-5a30a311d67e","Type":"ContainerStarted","Data":"21bb154822ec080a7815f8bb2c26dd42353a54c217fb1f68cd04045dbe6fe431"} Nov 03 15:08:48.943371 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:48.942971 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774c544669-f6n62" event={"ID":"bd5eb143-6167-431f-a09c-5a30a311d67e","Type":"ContainerStarted","Data":"7f0acfc84fb5fc598e1bf0246409362df7b196a91e05d6e06333a0e059659cf6"} Nov 03 15:08:49.064277 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:49.064229 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-774c544669-f6n62" podStartSLOduration=2.064214547 podStartE2EDuration="2.064214547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:49.062366983 +0000 UTC m=+36.493324811" watchObservedRunningTime="2026-11-03 15:08:49.064214547 +0000 UTC m=+36.495172389" Nov 03 15:08:50.926772 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:50.926741 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7bd85bcf57-n79t2" Nov 03 15:08:57.514051 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:57.513959 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:57.514051 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:57.514011 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:57.518792 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:57.518771 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:57.983562 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:57.983540 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-774c544669-f6n62" Nov 03 15:08:58.154899 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:08:58.154866 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77965c4cf-qdrhq"] Nov 03 15:09:00.008313 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:00.008278 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:09:00.008683 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:00.008362 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:09:08.700476 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:09:08.700390 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:09:20.013279 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:20.013252 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:09:20.017062 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:20.017039 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6699cbb5b6-xhr2d" Nov 03 15:09:23.175952 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.175895 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-77965c4cf-qdrhq" podUID="28bbf520-6347-4c53-992f-2d23875c02b2" containerName="console" containerID="cri-o://a23c86277b4125178fe45b32101b10577f51284a9d98264b54df70f6b0ce6a48" gracePeriod=15 Nov 03 15:09:23.452480 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.452421 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:09:23.516365 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.516344 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28bbf520-6347-4c53-992f-2d23875c02b2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28bbf520-6347-4c53-992f-2d23875c02b2-console-serving-cert\") pod \"28bbf520-6347-4c53-992f-2d23875c02b2\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " Nov 03 15:09:23.516477 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.516370 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28bbf520-6347-4c53-992f-2d23875c02b2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28bbf520-6347-4c53-992f-2d23875c02b2-console-oauth-config\") pod \"28bbf520-6347-4c53-992f-2d23875c02b2\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " Nov 03 15:09:23.516477 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.516412 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-console-config\" (UniqueName: \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-console-config\") pod \"28bbf520-6347-4c53-992f-2d23875c02b2\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " Nov 03 15:09:23.516477 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.516440 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bbf520-6347-4c53-992f-2d23875c02b2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/28bbf520-6347-4c53-992f-2d23875c02b2-tmp\") pod \"28bbf520-6347-4c53-992f-2d23875c02b2\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " Nov 03 15:09:23.516477 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.516457 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-oauth-serving-cert\") pod \"28bbf520-6347-4c53-992f-2d23875c02b2\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " Nov 03 15:09:23.516626 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.516517 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-service-ca\" (UniqueName: \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-service-ca\") pod \"28bbf520-6347-4c53-992f-2d23875c02b2\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " Nov 03 15:09:23.516626 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.516553 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28bbf520-6347-4c53-992f-2d23875c02b2-kube-api-access-b5976\" (UniqueName: \"kubernetes.io/projected/28bbf520-6347-4c53-992f-2d23875c02b2-kube-api-access-b5976\") pod \"28bbf520-6347-4c53-992f-2d23875c02b2\" (UID: \"28bbf520-6347-4c53-992f-2d23875c02b2\") " Nov 03 15:09:23.516805 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.516774 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bbf520-6347-4c53-992f-2d23875c02b2-tmp" pod "28bbf520-6347-4c53-992f-2d23875c02b2" (UID: "28bbf520-6347-4c53-992f-2d23875c02b2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:09:23.516901 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.516875 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-console-config" pod "28bbf520-6347-4c53-992f-2d23875c02b2" (UID: "28bbf520-6347-4c53-992f-2d23875c02b2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:09:23.520670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.517057 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-oauth-serving-cert" pod "28bbf520-6347-4c53-992f-2d23875c02b2" (UID: "28bbf520-6347-4c53-992f-2d23875c02b2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:09:23.520670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.517576 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-service-ca" pod "28bbf520-6347-4c53-992f-2d23875c02b2" (UID: "28bbf520-6347-4c53-992f-2d23875c02b2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:09:23.520849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.520751 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28bbf520-6347-4c53-992f-2d23875c02b2-kube-api-access-b5976" pod "28bbf520-6347-4c53-992f-2d23875c02b2" (UID: "28bbf520-6347-4c53-992f-2d23875c02b2"). InnerVolumeSpecName "kube-api-access-b5976". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:09:23.520849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.520831 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28bbf520-6347-4c53-992f-2d23875c02b2-console-serving-cert" pod "28bbf520-6347-4c53-992f-2d23875c02b2" (UID: "28bbf520-6347-4c53-992f-2d23875c02b2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:09:23.523293 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.523269 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28bbf520-6347-4c53-992f-2d23875c02b2-console-oauth-config" pod "28bbf520-6347-4c53-992f-2d23875c02b2" (UID: "28bbf520-6347-4c53-992f-2d23875c02b2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:09:23.617214 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.617184 4290 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-console-config\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:23.617214 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.617214 4290 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28bbf520-6347-4c53-992f-2d23875c02b2-tmp\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:23.617369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.617241 4290 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-oauth-serving-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:23.617369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.617250 4290 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28bbf520-6347-4c53-992f-2d23875c02b2-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:23.617369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.617259 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5976\" (UniqueName: \"kubernetes.io/projected/28bbf520-6347-4c53-992f-2d23875c02b2-kube-api-access-b5976\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:23.617369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.617275 4290 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28bbf520-6347-4c53-992f-2d23875c02b2-console-serving-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:23.617369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:23.617286 4290 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28bbf520-6347-4c53-992f-2d23875c02b2-console-oauth-config\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:24.074914 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:24.074879 4290 generic.go:356] "Generic (PLEG): container finished" podID="28bbf520-6347-4c53-992f-2d23875c02b2" containerID="a23c86277b4125178fe45b32101b10577f51284a9d98264b54df70f6b0ce6a48" exitCode=0 Nov 03 15:09:24.075088 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:24.074940 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77965c4cf-qdrhq" event={"ID":"28bbf520-6347-4c53-992f-2d23875c02b2","Type":"ContainerDied","Data":"a23c86277b4125178fe45b32101b10577f51284a9d98264b54df70f6b0ce6a48"} Nov 03 15:09:24.075088 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:24.074955 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77965c4cf-qdrhq" Nov 03 15:09:24.075088 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:24.074978 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77965c4cf-qdrhq" event={"ID":"28bbf520-6347-4c53-992f-2d23875c02b2","Type":"ContainerDied","Data":"f9de843d0e75874ea764beb56e94261f6173445cae95beec3ee462520f2301d3"} Nov 03 15:09:24.075088 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:24.074997 4290 scope.go:122] "RemoveContainer" containerID="a23c86277b4125178fe45b32101b10577f51284a9d98264b54df70f6b0ce6a48" Nov 03 15:09:24.083408 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:24.083393 4290 scope.go:122] "RemoveContainer" containerID="a23c86277b4125178fe45b32101b10577f51284a9d98264b54df70f6b0ce6a48" Nov 03 15:09:24.083659 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:09:24.083639 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a23c86277b4125178fe45b32101b10577f51284a9d98264b54df70f6b0ce6a48\": container with ID starting with a23c86277b4125178fe45b32101b10577f51284a9d98264b54df70f6b0ce6a48 not found: ID does not exist" containerID="a23c86277b4125178fe45b32101b10577f51284a9d98264b54df70f6b0ce6a48" Nov 03 15:09:24.083749 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:24.083672 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23c86277b4125178fe45b32101b10577f51284a9d98264b54df70f6b0ce6a48"} err="failed to get container status \"a23c86277b4125178fe45b32101b10577f51284a9d98264b54df70f6b0ce6a48\": rpc error: code = NotFound desc = could not find container \"a23c86277b4125178fe45b32101b10577f51284a9d98264b54df70f6b0ce6a48\": container with ID starting with a23c86277b4125178fe45b32101b10577f51284a9d98264b54df70f6b0ce6a48 not found: ID does not exist" Nov 03 15:09:24.240308 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:24.240276 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77965c4cf-qdrhq"] Nov 03 15:09:24.289934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:24.289909 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77965c4cf-qdrhq"] Nov 03 15:09:24.675657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:24.675626 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28bbf520-6347-4c53-992f-2d23875c02b2" path="/var/lib/kubelet/pods/28bbf520-6347-4c53-992f-2d23875c02b2/volumes" Nov 03 15:09:38.699608 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:09:38.699572 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:09:56.145557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.145526 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5669d4fc5f-sqssj"] Nov 03 15:09:56.146070 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.145885 4290 cpu_manager.go:414] "containerMap: removing container" podUID="28bbf520-6347-4c53-992f-2d23875c02b2" containerName="console" Nov 03 15:09:56.146070 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.145898 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28bbf520-6347-4c53-992f-2d23875c02b2" containerName="console" Nov 03 15:09:56.146070 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.145958 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="28bbf520-6347-4c53-992f-2d23875c02b2" containerName="console" Nov 03 15:09:56.147684 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.147670 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.250149 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.250113 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5669d4fc5f-sqssj"] Nov 03 15:09:56.266554 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.266527 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78fb751c-e0a9-42f8-8883-22d151c75714-console-config\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.266554 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.266556 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78fb751c-e0a9-42f8-8883-22d151c75714-tmp\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.266771 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.266585 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78fb751c-e0a9-42f8-8883-22d151c75714-service-ca\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.266771 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.266731 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78fb751c-e0a9-42f8-8883-22d151c75714-oauth-serving-cert\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.266842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.266780 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78fb751c-e0a9-42f8-8883-22d151c75714-console-serving-cert\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.266875 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.266834 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78fb751c-e0a9-42f8-8883-22d151c75714-trusted-ca-bundle\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.266875 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.266867 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx94w\" (UniqueName: \"kubernetes.io/projected/78fb751c-e0a9-42f8-8883-22d151c75714-kube-api-access-sx94w\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.266954 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.266937 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78fb751c-e0a9-42f8-8883-22d151c75714-console-oauth-config\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.367501 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.367465 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78fb751c-e0a9-42f8-8883-22d151c75714-console-oauth-config\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.367661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.367510 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78fb751c-e0a9-42f8-8883-22d151c75714-console-config\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.367661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.367528 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78fb751c-e0a9-42f8-8883-22d151c75714-tmp\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.367661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.367561 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78fb751c-e0a9-42f8-8883-22d151c75714-service-ca\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.367661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.367586 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78fb751c-e0a9-42f8-8883-22d151c75714-oauth-serving-cert\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.367661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.367605 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78fb751c-e0a9-42f8-8883-22d151c75714-console-serving-cert\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.367661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.367633 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78fb751c-e0a9-42f8-8883-22d151c75714-trusted-ca-bundle\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.367997 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.367663 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx94w\" (UniqueName: \"kubernetes.io/projected/78fb751c-e0a9-42f8-8883-22d151c75714-kube-api-access-sx94w\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.368223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.368195 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78fb751c-e0a9-42f8-8883-22d151c75714-tmp\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.368530 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.368507 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78fb751c-e0a9-42f8-8883-22d151c75714-service-ca\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.368606 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.368549 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78fb751c-e0a9-42f8-8883-22d151c75714-console-config\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.368606 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.368564 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78fb751c-e0a9-42f8-8883-22d151c75714-oauth-serving-cert\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.368724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.368688 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78fb751c-e0a9-42f8-8883-22d151c75714-trusted-ca-bundle\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.370162 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.370138 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78fb751c-e0a9-42f8-8883-22d151c75714-console-oauth-config\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.370268 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.370251 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78fb751c-e0a9-42f8-8883-22d151c75714-console-serving-cert\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.429246 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.429185 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx94w\" (UniqueName: \"kubernetes.io/projected/78fb751c-e0a9-42f8-8883-22d151c75714-kube-api-access-sx94w\") pod \"console-5669d4fc5f-sqssj\" (UID: \"78fb751c-e0a9-42f8-8883-22d151c75714\") " pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.458090 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.458062 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:09:56.831592 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:56.831486 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5669d4fc5f-sqssj"] Nov 03 15:09:56.834167 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:09:56.834144 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78fb751c_e0a9_42f8_8883_22d151c75714.slice/crio-6aefc46901a45d395557cfbeb74040a44bcf31c98ffa6f24ed2e5f774e2ebbab WatchSource:0}: Error finding container 6aefc46901a45d395557cfbeb74040a44bcf31c98ffa6f24ed2e5f774e2ebbab: Status 404 returned error can't find the container with id 6aefc46901a45d395557cfbeb74040a44bcf31c98ffa6f24ed2e5f774e2ebbab Nov 03 15:09:57.198026 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:57.197988 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5669d4fc5f-sqssj" event={"ID":"78fb751c-e0a9-42f8-8883-22d151c75714","Type":"ContainerStarted","Data":"f3e397f40771892631bbfe23d4fafde9eae9fc2787dfc849fba06f51c5b3c173"} Nov 03 15:09:57.198463 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:57.198059 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5669d4fc5f-sqssj" event={"ID":"78fb751c-e0a9-42f8-8883-22d151c75714","Type":"ContainerStarted","Data":"6aefc46901a45d395557cfbeb74040a44bcf31c98ffa6f24ed2e5f774e2ebbab"} Nov 03 15:09:57.317900 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:09:57.317847 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5669d4fc5f-sqssj" podStartSLOduration=1.317833335 podStartE2EDuration="1.317833335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:09:57.317231912 +0000 UTC m=+104.748189719" watchObservedRunningTime="2026-11-03 15:09:57.317833335 +0000 UTC m=+104.748791141" Nov 03 15:10:01.394710 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:01.394676 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 15:10:01.395117 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:01.395009 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="alertmanager" containerID="cri-o://cfac535e02b408db6bf6b44bdb81cfa6381b46c9898af2345d25f9cf4d671151" gracePeriod=120 Nov 03 15:10:01.395117 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:01.395075 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="kube-rbac-proxy-metric" containerID="cri-o://2014885e775cc7842e0578ce1525a278cb9f7bb0e6cc6b6f8ecbb01f46e3a54d" gracePeriod=120 Nov 03 15:10:01.395117 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:01.395105 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="kube-rbac-proxy" containerID="cri-o://773b75d1d36e4e8cee6bd7438648b6228ebae6342047d29c40ec32ac12d07f74" gracePeriod=120 Nov 03 15:10:01.395292 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:01.395136 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="config-reloader" containerID="cri-o://fc07a8baaa6b80d200bb63afe714864fec646bd5a5570fa6d040525662b120ce" gracePeriod=120 Nov 03 15:10:01.395292 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:01.395084 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="kube-rbac-proxy-web" containerID="cri-o://142d1ac12130abd4b535299fd06c075e44a81605826754153e1dfaf775f49b57" gracePeriod=120 Nov 03 15:10:01.395292 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:01.395201 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="prom-label-proxy" containerID="cri-o://bfbfa0fe467b43cf3f24648a566983ac75751a96767de061fbb082f589c1254b" gracePeriod=120 Nov 03 15:10:02.220624 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.220591 4290 generic.go:356] "Generic (PLEG): container finished" podID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerID="bfbfa0fe467b43cf3f24648a566983ac75751a96767de061fbb082f589c1254b" exitCode=0 Nov 03 15:10:02.220624 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.220617 4290 generic.go:356] "Generic (PLEG): container finished" podID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerID="773b75d1d36e4e8cee6bd7438648b6228ebae6342047d29c40ec32ac12d07f74" exitCode=0 Nov 03 15:10:02.220624 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.220623 4290 generic.go:356] "Generic (PLEG): container finished" podID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerID="142d1ac12130abd4b535299fd06c075e44a81605826754153e1dfaf775f49b57" exitCode=0 Nov 03 15:10:02.220624 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.220628 4290 generic.go:356] "Generic (PLEG): container finished" podID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerID="fc07a8baaa6b80d200bb63afe714864fec646bd5a5570fa6d040525662b120ce" exitCode=0 Nov 03 15:10:02.220624 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.220633 4290 generic.go:356] "Generic (PLEG): container finished" podID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerID="cfac535e02b408db6bf6b44bdb81cfa6381b46c9898af2345d25f9cf4d671151" exitCode=0 Nov 03 15:10:02.221076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.220666 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5c76303-7f6d-467e-8e89-0d1a925afd02","Type":"ContainerDied","Data":"bfbfa0fe467b43cf3f24648a566983ac75751a96767de061fbb082f589c1254b"} Nov 03 15:10:02.221076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.220779 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5c76303-7f6d-467e-8e89-0d1a925afd02","Type":"ContainerDied","Data":"773b75d1d36e4e8cee6bd7438648b6228ebae6342047d29c40ec32ac12d07f74"} Nov 03 15:10:02.221076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.220808 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5c76303-7f6d-467e-8e89-0d1a925afd02","Type":"ContainerDied","Data":"142d1ac12130abd4b535299fd06c075e44a81605826754153e1dfaf775f49b57"} Nov 03 15:10:02.221076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.220833 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5c76303-7f6d-467e-8e89-0d1a925afd02","Type":"ContainerDied","Data":"fc07a8baaa6b80d200bb63afe714864fec646bd5a5570fa6d040525662b120ce"} Nov 03 15:10:02.221076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.220858 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5c76303-7f6d-467e-8e89-0d1a925afd02","Type":"ContainerDied","Data":"cfac535e02b408db6bf6b44bdb81cfa6381b46c9898af2345d25f9cf4d671151"} Nov 03 15:10:02.645043 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.645021 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:02.824937 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.824865 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c76303-7f6d-467e-8e89-0d1a925afd02-kube-api-access-kfdj6\" (UniqueName: \"kubernetes.io/projected/a5c76303-7f6d-467e-8e89-0d1a925afd02-kube-api-access-kfdj6\") pod \"a5c76303-7f6d-467e-8e89-0d1a925afd02\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " Nov 03 15:10:02.825096 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.824901 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy-web\") pod \"a5c76303-7f6d-467e-8e89-0d1a925afd02\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " Nov 03 15:10:02.825096 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.825001 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-config-volume\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-config-volume\") pod \"a5c76303-7f6d-467e-8e89-0d1a925afd02\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " Nov 03 15:10:02.825096 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.825033 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a5c76303-7f6d-467e-8e89-0d1a925afd02\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " Nov 03 15:10:02.825096 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.825060 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c76303-7f6d-467e-8e89-0d1a925afd02-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5c76303-7f6d-467e-8e89-0d1a925afd02-alertmanager-trusted-ca-bundle\") pod \"a5c76303-7f6d-467e-8e89-0d1a925afd02\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " Nov 03 15:10:02.825319 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.825097 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c76303-7f6d-467e-8e89-0d1a925afd02-tls-assets\" (UniqueName: \"kubernetes.io/projected/a5c76303-7f6d-467e-8e89-0d1a925afd02-tls-assets\") pod \"a5c76303-7f6d-467e-8e89-0d1a925afd02\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " Nov 03 15:10:02.825319 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.825128 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy\") pod \"a5c76303-7f6d-467e-8e89-0d1a925afd02\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " Nov 03 15:10:02.825319 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.825165 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-web-config\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-web-config\") pod \"a5c76303-7f6d-467e-8e89-0d1a925afd02\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " Nov 03 15:10:02.825319 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.825240 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c76303-7f6d-467e-8e89-0d1a925afd02-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5c76303-7f6d-467e-8e89-0d1a925afd02-config-out\") pod \"a5c76303-7f6d-467e-8e89-0d1a925afd02\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " Nov 03 15:10:02.825319 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.825285 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c76303-7f6d-467e-8e89-0d1a925afd02-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a5c76303-7f6d-467e-8e89-0d1a925afd02-alertmanager-main-db\") pod \"a5c76303-7f6d-467e-8e89-0d1a925afd02\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " Nov 03 15:10:02.825319 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.825315 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-cluster-tls-config\") pod \"a5c76303-7f6d-467e-8e89-0d1a925afd02\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " Nov 03 15:10:02.825622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.825362 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c76303-7f6d-467e-8e89-0d1a925afd02-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5c76303-7f6d-467e-8e89-0d1a925afd02-metrics-client-ca\") pod \"a5c76303-7f6d-467e-8e89-0d1a925afd02\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " Nov 03 15:10:02.825622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.825392 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-main-tls\") pod \"a5c76303-7f6d-467e-8e89-0d1a925afd02\" (UID: \"a5c76303-7f6d-467e-8e89-0d1a925afd02\") " Nov 03 15:10:02.825622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.825578 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c76303-7f6d-467e-8e89-0d1a925afd02-alertmanager-trusted-ca-bundle" pod "a5c76303-7f6d-467e-8e89-0d1a925afd02" (UID: "a5c76303-7f6d-467e-8e89-0d1a925afd02"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:10:02.825973 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.825877 4290 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5c76303-7f6d-467e-8e89-0d1a925afd02-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:02.826444 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.826119 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c76303-7f6d-467e-8e89-0d1a925afd02-alertmanager-main-db" pod "a5c76303-7f6d-467e-8e89-0d1a925afd02" (UID: "a5c76303-7f6d-467e-8e89-0d1a925afd02"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:10:02.826444 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.826437 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c76303-7f6d-467e-8e89-0d1a925afd02-metrics-client-ca" pod "a5c76303-7f6d-467e-8e89-0d1a925afd02" (UID: "a5c76303-7f6d-467e-8e89-0d1a925afd02"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:10:02.828860 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.828831 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy-metric" pod "a5c76303-7f6d-467e-8e89-0d1a925afd02" (UID: "a5c76303-7f6d-467e-8e89-0d1a925afd02"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:02.828972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.828921 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-config-volume" pod "a5c76303-7f6d-467e-8e89-0d1a925afd02" (UID: "a5c76303-7f6d-467e-8e89-0d1a925afd02"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:02.829244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.829220 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c76303-7f6d-467e-8e89-0d1a925afd02-kube-api-access-kfdj6" pod "a5c76303-7f6d-467e-8e89-0d1a925afd02" (UID: "a5c76303-7f6d-467e-8e89-0d1a925afd02"). InnerVolumeSpecName "kube-api-access-kfdj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:10:02.829363 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.829334 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy-web" pod "a5c76303-7f6d-467e-8e89-0d1a925afd02" (UID: "a5c76303-7f6d-467e-8e89-0d1a925afd02"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:02.829655 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.829626 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c76303-7f6d-467e-8e89-0d1a925afd02-config-out" pod "a5c76303-7f6d-467e-8e89-0d1a925afd02" (UID: "a5c76303-7f6d-467e-8e89-0d1a925afd02"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:10:02.829783 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.829680 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-main-tls" pod "a5c76303-7f6d-467e-8e89-0d1a925afd02" (UID: "a5c76303-7f6d-467e-8e89-0d1a925afd02"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:02.829783 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.829747 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy" pod "a5c76303-7f6d-467e-8e89-0d1a925afd02" (UID: "a5c76303-7f6d-467e-8e89-0d1a925afd02"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:02.829862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.829812 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c76303-7f6d-467e-8e89-0d1a925afd02-tls-assets" pod "a5c76303-7f6d-467e-8e89-0d1a925afd02" (UID: "a5c76303-7f6d-467e-8e89-0d1a925afd02"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:10:02.834177 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.834140 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-cluster-tls-config" pod "a5c76303-7f6d-467e-8e89-0d1a925afd02" (UID: "a5c76303-7f6d-467e-8e89-0d1a925afd02"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:02.840255 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.840235 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-web-config" pod "a5c76303-7f6d-467e-8e89-0d1a925afd02" (UID: "a5c76303-7f6d-467e-8e89-0d1a925afd02"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:02.927004 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.926973 4290 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5c76303-7f6d-467e-8e89-0d1a925afd02-metrics-client-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:02.927004 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.927007 4290 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-main-tls\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:02.927182 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.927022 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfdj6\" (UniqueName: \"kubernetes.io/projected/a5c76303-7f6d-467e-8e89-0d1a925afd02-kube-api-access-kfdj6\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:02.927182 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.927038 4290 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:02.927182 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.927051 4290 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:02.927182 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.927066 4290 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:02.927182 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.927082 4290 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5c76303-7f6d-467e-8e89-0d1a925afd02-tls-assets\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:02.927182 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.927096 4290 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:02.927182 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.927110 4290 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-web-config\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:02.927182 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.927123 4290 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5c76303-7f6d-467e-8e89-0d1a925afd02-config-out\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:02.927182 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.927136 4290 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a5c76303-7f6d-467e-8e89-0d1a925afd02-alertmanager-main-db\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:02.927182 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:02.927149 4290 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a5c76303-7f6d-467e-8e89-0d1a925afd02-cluster-tls-config\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:03.227810 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.227776 4290 generic.go:356] "Generic (PLEG): container finished" podID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerID="2014885e775cc7842e0578ce1525a278cb9f7bb0e6cc6b6f8ecbb01f46e3a54d" exitCode=0 Nov 03 15:10:03.227976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.227857 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.227976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.227854 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5c76303-7f6d-467e-8e89-0d1a925afd02","Type":"ContainerDied","Data":"2014885e775cc7842e0578ce1525a278cb9f7bb0e6cc6b6f8ecbb01f46e3a54d"} Nov 03 15:10:03.228062 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.227989 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5c76303-7f6d-467e-8e89-0d1a925afd02","Type":"ContainerDied","Data":"b81cf052744fdb489cec35d43911c7b1be4c8ae3749a4257cc3e90ab50d27b1e"} Nov 03 15:10:03.228062 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.228021 4290 scope.go:122] "RemoveContainer" containerID="bfbfa0fe467b43cf3f24648a566983ac75751a96767de061fbb082f589c1254b" Nov 03 15:10:03.236772 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.236751 4290 scope.go:122] "RemoveContainer" containerID="2014885e775cc7842e0578ce1525a278cb9f7bb0e6cc6b6f8ecbb01f46e3a54d" Nov 03 15:10:03.245576 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.245560 4290 scope.go:122] "RemoveContainer" containerID="773b75d1d36e4e8cee6bd7438648b6228ebae6342047d29c40ec32ac12d07f74" Nov 03 15:10:03.253170 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.253155 4290 scope.go:122] "RemoveContainer" containerID="142d1ac12130abd4b535299fd06c075e44a81605826754153e1dfaf775f49b57" Nov 03 15:10:03.259830 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.259813 4290 scope.go:122] "RemoveContainer" containerID="fc07a8baaa6b80d200bb63afe714864fec646bd5a5570fa6d040525662b120ce" Nov 03 15:10:03.266932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.266911 4290 scope.go:122] "RemoveContainer" containerID="cfac535e02b408db6bf6b44bdb81cfa6381b46c9898af2345d25f9cf4d671151" Nov 03 15:10:03.274001 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.273982 4290 scope.go:122] "RemoveContainer" containerID="1519fb1aa0e1911f44812d9409c6c5b0d2989547766266e99b9ded37302d1afe" Nov 03 15:10:03.280849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.280809 4290 scope.go:122] "RemoveContainer" containerID="bfbfa0fe467b43cf3f24648a566983ac75751a96767de061fbb082f589c1254b" Nov 03 15:10:03.281179 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:10:03.281143 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfbfa0fe467b43cf3f24648a566983ac75751a96767de061fbb082f589c1254b\": container with ID starting with bfbfa0fe467b43cf3f24648a566983ac75751a96767de061fbb082f589c1254b not found: ID does not exist" containerID="bfbfa0fe467b43cf3f24648a566983ac75751a96767de061fbb082f589c1254b" Nov 03 15:10:03.281260 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.281174 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfbfa0fe467b43cf3f24648a566983ac75751a96767de061fbb082f589c1254b"} err="failed to get container status \"bfbfa0fe467b43cf3f24648a566983ac75751a96767de061fbb082f589c1254b\": rpc error: code = NotFound desc = could not find container \"bfbfa0fe467b43cf3f24648a566983ac75751a96767de061fbb082f589c1254b\": container with ID starting with bfbfa0fe467b43cf3f24648a566983ac75751a96767de061fbb082f589c1254b not found: ID does not exist" Nov 03 15:10:03.281260 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.281193 4290 scope.go:122] "RemoveContainer" containerID="2014885e775cc7842e0578ce1525a278cb9f7bb0e6cc6b6f8ecbb01f46e3a54d" Nov 03 15:10:03.281506 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:10:03.281474 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2014885e775cc7842e0578ce1525a278cb9f7bb0e6cc6b6f8ecbb01f46e3a54d\": container with ID starting with 2014885e775cc7842e0578ce1525a278cb9f7bb0e6cc6b6f8ecbb01f46e3a54d not found: ID does not exist" containerID="2014885e775cc7842e0578ce1525a278cb9f7bb0e6cc6b6f8ecbb01f46e3a54d" Nov 03 15:10:03.281544 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.281517 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2014885e775cc7842e0578ce1525a278cb9f7bb0e6cc6b6f8ecbb01f46e3a54d"} err="failed to get container status \"2014885e775cc7842e0578ce1525a278cb9f7bb0e6cc6b6f8ecbb01f46e3a54d\": rpc error: code = NotFound desc = could not find container \"2014885e775cc7842e0578ce1525a278cb9f7bb0e6cc6b6f8ecbb01f46e3a54d\": container with ID starting with 2014885e775cc7842e0578ce1525a278cb9f7bb0e6cc6b6f8ecbb01f46e3a54d not found: ID does not exist" Nov 03 15:10:03.281544 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.281539 4290 scope.go:122] "RemoveContainer" containerID="773b75d1d36e4e8cee6bd7438648b6228ebae6342047d29c40ec32ac12d07f74" Nov 03 15:10:03.281829 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:10:03.281813 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773b75d1d36e4e8cee6bd7438648b6228ebae6342047d29c40ec32ac12d07f74\": container with ID starting with 773b75d1d36e4e8cee6bd7438648b6228ebae6342047d29c40ec32ac12d07f74 not found: ID does not exist" containerID="773b75d1d36e4e8cee6bd7438648b6228ebae6342047d29c40ec32ac12d07f74" Nov 03 15:10:03.281892 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.281834 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773b75d1d36e4e8cee6bd7438648b6228ebae6342047d29c40ec32ac12d07f74"} err="failed to get container status \"773b75d1d36e4e8cee6bd7438648b6228ebae6342047d29c40ec32ac12d07f74\": rpc error: code = NotFound desc = could not find container \"773b75d1d36e4e8cee6bd7438648b6228ebae6342047d29c40ec32ac12d07f74\": container with ID starting with 773b75d1d36e4e8cee6bd7438648b6228ebae6342047d29c40ec32ac12d07f74 not found: ID does not exist" Nov 03 15:10:03.281892 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.281848 4290 scope.go:122] "RemoveContainer" containerID="142d1ac12130abd4b535299fd06c075e44a81605826754153e1dfaf775f49b57" Nov 03 15:10:03.282099 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:10:03.282085 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"142d1ac12130abd4b535299fd06c075e44a81605826754153e1dfaf775f49b57\": container with ID starting with 142d1ac12130abd4b535299fd06c075e44a81605826754153e1dfaf775f49b57 not found: ID does not exist" containerID="142d1ac12130abd4b535299fd06c075e44a81605826754153e1dfaf775f49b57" Nov 03 15:10:03.282139 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.282101 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"142d1ac12130abd4b535299fd06c075e44a81605826754153e1dfaf775f49b57"} err="failed to get container status \"142d1ac12130abd4b535299fd06c075e44a81605826754153e1dfaf775f49b57\": rpc error: code = NotFound desc = could not find container \"142d1ac12130abd4b535299fd06c075e44a81605826754153e1dfaf775f49b57\": container with ID starting with 142d1ac12130abd4b535299fd06c075e44a81605826754153e1dfaf775f49b57 not found: ID does not exist" Nov 03 15:10:03.282139 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.282113 4290 scope.go:122] "RemoveContainer" containerID="fc07a8baaa6b80d200bb63afe714864fec646bd5a5570fa6d040525662b120ce" Nov 03 15:10:03.282351 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:10:03.282323 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc07a8baaa6b80d200bb63afe714864fec646bd5a5570fa6d040525662b120ce\": container with ID starting with fc07a8baaa6b80d200bb63afe714864fec646bd5a5570fa6d040525662b120ce not found: ID does not exist" containerID="fc07a8baaa6b80d200bb63afe714864fec646bd5a5570fa6d040525662b120ce" Nov 03 15:10:03.282399 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.282354 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc07a8baaa6b80d200bb63afe714864fec646bd5a5570fa6d040525662b120ce"} err="failed to get container status \"fc07a8baaa6b80d200bb63afe714864fec646bd5a5570fa6d040525662b120ce\": rpc error: code = NotFound desc = could not find container \"fc07a8baaa6b80d200bb63afe714864fec646bd5a5570fa6d040525662b120ce\": container with ID starting with fc07a8baaa6b80d200bb63afe714864fec646bd5a5570fa6d040525662b120ce not found: ID does not exist" Nov 03 15:10:03.282399 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.282368 4290 scope.go:122] "RemoveContainer" containerID="cfac535e02b408db6bf6b44bdb81cfa6381b46c9898af2345d25f9cf4d671151" Nov 03 15:10:03.282628 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:10:03.282611 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfac535e02b408db6bf6b44bdb81cfa6381b46c9898af2345d25f9cf4d671151\": container with ID starting with cfac535e02b408db6bf6b44bdb81cfa6381b46c9898af2345d25f9cf4d671151 not found: ID does not exist" containerID="cfac535e02b408db6bf6b44bdb81cfa6381b46c9898af2345d25f9cf4d671151" Nov 03 15:10:03.282675 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.282640 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfac535e02b408db6bf6b44bdb81cfa6381b46c9898af2345d25f9cf4d671151"} err="failed to get container status \"cfac535e02b408db6bf6b44bdb81cfa6381b46c9898af2345d25f9cf4d671151\": rpc error: code = NotFound desc = could not find container \"cfac535e02b408db6bf6b44bdb81cfa6381b46c9898af2345d25f9cf4d671151\": container with ID starting with cfac535e02b408db6bf6b44bdb81cfa6381b46c9898af2345d25f9cf4d671151 not found: ID does not exist" Nov 03 15:10:03.282675 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.282659 4290 scope.go:122] "RemoveContainer" containerID="1519fb1aa0e1911f44812d9409c6c5b0d2989547766266e99b9ded37302d1afe" Nov 03 15:10:03.283007 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:10:03.282991 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1519fb1aa0e1911f44812d9409c6c5b0d2989547766266e99b9ded37302d1afe\": container with ID starting with 1519fb1aa0e1911f44812d9409c6c5b0d2989547766266e99b9ded37302d1afe not found: ID does not exist" containerID="1519fb1aa0e1911f44812d9409c6c5b0d2989547766266e99b9ded37302d1afe" Nov 03 15:10:03.283062 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.283010 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1519fb1aa0e1911f44812d9409c6c5b0d2989547766266e99b9ded37302d1afe"} err="failed to get container status \"1519fb1aa0e1911f44812d9409c6c5b0d2989547766266e99b9ded37302d1afe\": rpc error: code = NotFound desc = could not find container \"1519fb1aa0e1911f44812d9409c6c5b0d2989547766266e99b9ded37302d1afe\": container with ID starting with 1519fb1aa0e1911f44812d9409c6c5b0d2989547766266e99b9ded37302d1afe not found: ID does not exist" Nov 03 15:10:03.396888 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.396861 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 15:10:03.447102 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.447075 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 15:10:03.448036 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448019 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 15:10:03.448352 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448327 4290 cpu_manager.go:414] "containerMap: removing container" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="kube-rbac-proxy-metric" Nov 03 15:10:03.448352 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448347 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="kube-rbac-proxy-metric" Nov 03 15:10:03.448431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448362 4290 cpu_manager.go:414] "containerMap: removing container" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="kube-rbac-proxy-web" Nov 03 15:10:03.448431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448367 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="kube-rbac-proxy-web" Nov 03 15:10:03.448431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448380 4290 cpu_manager.go:414] "containerMap: removing container" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="kube-rbac-proxy" Nov 03 15:10:03.448431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448385 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="kube-rbac-proxy" Nov 03 15:10:03.448431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448400 4290 cpu_manager.go:414] "containerMap: removing container" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="alertmanager" Nov 03 15:10:03.448431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448404 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="alertmanager" Nov 03 15:10:03.448431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448412 4290 cpu_manager.go:414] "containerMap: removing container" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="config-reloader" Nov 03 15:10:03.448431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448416 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="config-reloader" Nov 03 15:10:03.448431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448424 4290 cpu_manager.go:414] "containerMap: removing container" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="prom-label-proxy" Nov 03 15:10:03.448431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448429 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="prom-label-proxy" Nov 03 15:10:03.448720 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448437 4290 cpu_manager.go:414] "containerMap: removing container" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="init-config-reloader" Nov 03 15:10:03.448720 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448441 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="init-config-reloader" Nov 03 15:10:03.448720 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448488 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="config-reloader" Nov 03 15:10:03.448720 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448495 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="prom-label-proxy" Nov 03 15:10:03.448720 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448502 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="kube-rbac-proxy-metric" Nov 03 15:10:03.448720 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448508 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="kube-rbac-proxy-web" Nov 03 15:10:03.448720 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448514 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="kube-rbac-proxy" Nov 03 15:10:03.448720 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.448522 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" containerName="alertmanager" Nov 03 15:10:03.451660 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.451642 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.531214 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.531190 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.531350 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.531241 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.531350 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.531341 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-config-volume\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.531452 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.531369 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.531452 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.531386 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.531452 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.531403 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-web-config\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.531609 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.531472 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.531609 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.531567 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.531609 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.531596 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.531750 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.531631 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-config-out\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.531750 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.531694 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.531811 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.531757 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s89sj\" (UniqueName: \"kubernetes.io/projected/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-kube-api-access-s89sj\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.531811 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.531781 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.554460 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.554437 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 15:10:03.554460 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.554452 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 15:10:03.554600 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.554452 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 15:10:03.554600 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.554492 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 15:10:03.554600 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.554498 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 15:10:03.554600 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.554486 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 15:10:03.554600 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.554511 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 15:10:03.554991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.554964 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 15:10:03.555114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.555085 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jxtqz\"" Nov 03 15:10:03.558179 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.557802 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 15:10:03.569914 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.569886 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 15:10:03.632883 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.632852 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.633094 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.632890 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.633094 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.632910 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.633094 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.632938 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-config-out\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.633094 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.633076 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.633289 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.633120 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s89sj\" (UniqueName: \"kubernetes.io/projected/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-kube-api-access-s89sj\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.633289 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.633160 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.633289 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.633207 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.633289 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.633265 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.633492 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.633296 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-config-volume\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.633492 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.633327 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.633492 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.633355 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.633492 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.633380 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-web-config\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.633800 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.633765 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.633958 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.633937 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.634025 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.633977 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.636384 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.636358 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.636490 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.636362 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-config-out\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.636623 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.636596 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-config-volume\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.637049 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.637029 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-web-config\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.637134 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.637051 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.637197 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.637171 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.637232 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.637203 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.637370 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.637349 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.638633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.638610 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.707984 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.707954 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s89sj\" (UniqueName: \"kubernetes.io/projected/ceb2931e-109a-4cec-ac2e-3b2f18e5de55-kube-api-access-s89sj\") pod \"alertmanager-main-0\" (UID: \"ceb2931e-109a-4cec-ac2e-3b2f18e5de55\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:03.768139 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:03.768082 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 15:10:04.199052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:04.199024 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 15:10:04.201720 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:10:04.201663 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb2931e_109a_4cec_ac2e_3b2f18e5de55.slice/crio-15276aa7b3aa3cdc04d93e8cf0083adc486571eb2614d50701aa6c922745e39a WatchSource:0}: Error finding container 15276aa7b3aa3cdc04d93e8cf0083adc486571eb2614d50701aa6c922745e39a: Status 404 returned error can't find the container with id 15276aa7b3aa3cdc04d93e8cf0083adc486571eb2614d50701aa6c922745e39a Nov 03 15:10:04.232716 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:04.232664 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb2931e-109a-4cec-ac2e-3b2f18e5de55","Type":"ContainerStarted","Data":"15276aa7b3aa3cdc04d93e8cf0083adc486571eb2614d50701aa6c922745e39a"} Nov 03 15:10:04.675492 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:04.675465 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c76303-7f6d-467e-8e89-0d1a925afd02" path="/var/lib/kubelet/pods/a5c76303-7f6d-467e-8e89-0d1a925afd02/volumes" Nov 03 15:10:05.238628 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:05.238592 4290 generic.go:356] "Generic (PLEG): container finished" podID="ceb2931e-109a-4cec-ac2e-3b2f18e5de55" containerID="e09b0ba49270c6f0ade3d261ab1f1aafb721de754f338210c8fdc6bb09138d9a" exitCode=0 Nov 03 15:10:05.239115 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:05.238659 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb2931e-109a-4cec-ac2e-3b2f18e5de55","Type":"ContainerDied","Data":"e09b0ba49270c6f0ade3d261ab1f1aafb721de754f338210c8fdc6bb09138d9a"} Nov 03 15:10:06.247516 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:06.247474 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb2931e-109a-4cec-ac2e-3b2f18e5de55","Type":"ContainerStarted","Data":"296e9f5dd2ce5e6b7b24d79d7a154d77d3b5c60351983055484ae22225b3dc1c"} Nov 03 15:10:06.248052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:06.247541 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb2931e-109a-4cec-ac2e-3b2f18e5de55","Type":"ContainerStarted","Data":"6a33a3a4a0c74bb15344a8cba3ae0820da07bc119fb3f0025508aecf5ec4a80d"} Nov 03 15:10:06.248052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:06.247572 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb2931e-109a-4cec-ac2e-3b2f18e5de55","Type":"ContainerStarted","Data":"18a272e9b1df0a6e78a3d683531fbad75f1a2426d66d97eafa132ce4c0f43964"} Nov 03 15:10:06.248052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:06.247599 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb2931e-109a-4cec-ac2e-3b2f18e5de55","Type":"ContainerStarted","Data":"2ae4718bf31259cf94a599ff5fc24ece8d0476cdd03249684d3dc115ddeba267"} Nov 03 15:10:06.248052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:06.247622 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb2931e-109a-4cec-ac2e-3b2f18e5de55","Type":"ContainerStarted","Data":"e2037f58a0c2264b00166faecfcea9b2f52d0b008e1e61be1f0dce04680bb75c"} Nov 03 15:10:06.248052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:06.247656 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb2931e-109a-4cec-ac2e-3b2f18e5de55","Type":"ContainerStarted","Data":"5a36ef867e8692466b1b08563e2f3b6c0d2286aa7435e9a760a0e70e98a77298"} Nov 03 15:10:06.380740 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:06.379981 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.370057739 podStartE2EDuration="3.379964902s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.907163ms" isStatefulPod=false podCreationTimestamp="2026-11-03 15:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:10:06.378307206 +0000 UTC m=+113.809265013" watchObservedRunningTime="2026-11-03 15:10:06.379964902 +0000 UTC m=+113.810922707" Nov 03 15:10:06.458957 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:06.458928 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:10:06.459114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:06.458985 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:10:06.463636 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:06.463613 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:10:07.255693 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:07.255666 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5669d4fc5f-sqssj" Nov 03 15:10:07.413123 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:07.413093 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-774c544669-f6n62"] Nov 03 15:10:08.700959 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:10:08.700883 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:10:32.432682 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.432581 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-774c544669-f6n62" podUID="bd5eb143-6167-431f-a09c-5a30a311d67e" containerName="console" containerID="cri-o://21bb154822ec080a7815f8bb2c26dd42353a54c217fb1f68cd04045dbe6fe431" gracePeriod=15 Nov 03 15:10:32.677850 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.677825 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774c544669-f6n62" Nov 03 15:10:32.761716 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.761631 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-oauth-serving-cert\") pod \"bd5eb143-6167-431f-a09c-5a30a311d67e\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " Nov 03 15:10:32.761716 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.761677 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd5eb143-6167-431f-a09c-5a30a311d67e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd5eb143-6167-431f-a09c-5a30a311d67e-console-serving-cert\") pod \"bd5eb143-6167-431f-a09c-5a30a311d67e\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " Nov 03 15:10:32.761927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.761727 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd5eb143-6167-431f-a09c-5a30a311d67e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd5eb143-6167-431f-a09c-5a30a311d67e-console-oauth-config\") pod \"bd5eb143-6167-431f-a09c-5a30a311d67e\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " Nov 03 15:10:32.761927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.761796 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd5eb143-6167-431f-a09c-5a30a311d67e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd5eb143-6167-431f-a09c-5a30a311d67e-tmp\") pod \"bd5eb143-6167-431f-a09c-5a30a311d67e\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " Nov 03 15:10:32.761927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.761836 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-trusted-ca-bundle\") pod \"bd5eb143-6167-431f-a09c-5a30a311d67e\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " Nov 03 15:10:32.761927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.761880 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-service-ca\") pod \"bd5eb143-6167-431f-a09c-5a30a311d67e\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " Nov 03 15:10:32.761927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.761916 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd5eb143-6167-431f-a09c-5a30a311d67e-kube-api-access-5f8ct\" (UniqueName: \"kubernetes.io/projected/bd5eb143-6167-431f-a09c-5a30a311d67e-kube-api-access-5f8ct\") pod \"bd5eb143-6167-431f-a09c-5a30a311d67e\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " Nov 03 15:10:32.762180 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.761964 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-console-config\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-console-config\") pod \"bd5eb143-6167-431f-a09c-5a30a311d67e\" (UID: \"bd5eb143-6167-431f-a09c-5a30a311d67e\") " Nov 03 15:10:32.762180 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.762099 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-oauth-serving-cert" pod "bd5eb143-6167-431f-a09c-5a30a311d67e" (UID: "bd5eb143-6167-431f-a09c-5a30a311d67e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:10:32.762180 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.762130 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd5eb143-6167-431f-a09c-5a30a311d67e-tmp" pod "bd5eb143-6167-431f-a09c-5a30a311d67e" (UID: "bd5eb143-6167-431f-a09c-5a30a311d67e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:10:32.762332 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.762243 4290 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd5eb143-6167-431f-a09c-5a30a311d67e-tmp\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:32.762332 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.762260 4290 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-oauth-serving-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:32.762762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.762620 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-service-ca" pod "bd5eb143-6167-431f-a09c-5a30a311d67e" (UID: "bd5eb143-6167-431f-a09c-5a30a311d67e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:10:32.762865 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.762840 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-console-config" pod "bd5eb143-6167-431f-a09c-5a30a311d67e" (UID: "bd5eb143-6167-431f-a09c-5a30a311d67e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:10:32.762914 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.762871 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-trusted-ca-bundle" pod "bd5eb143-6167-431f-a09c-5a30a311d67e" (UID: "bd5eb143-6167-431f-a09c-5a30a311d67e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:10:32.764201 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.764170 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd5eb143-6167-431f-a09c-5a30a311d67e-console-serving-cert" pod "bd5eb143-6167-431f-a09c-5a30a311d67e" (UID: "bd5eb143-6167-431f-a09c-5a30a311d67e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:32.764540 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.764519 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd5eb143-6167-431f-a09c-5a30a311d67e-console-oauth-config" pod "bd5eb143-6167-431f-a09c-5a30a311d67e" (UID: "bd5eb143-6167-431f-a09c-5a30a311d67e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:32.764776 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.764751 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5eb143-6167-431f-a09c-5a30a311d67e-kube-api-access-5f8ct" pod "bd5eb143-6167-431f-a09c-5a30a311d67e" (UID: "bd5eb143-6167-431f-a09c-5a30a311d67e"). InnerVolumeSpecName "kube-api-access-5f8ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:10:32.862575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.862548 4290 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd5eb143-6167-431f-a09c-5a30a311d67e-console-serving-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:32.862575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.862567 4290 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd5eb143-6167-431f-a09c-5a30a311d67e-console-oauth-config\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:32.862575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.862576 4290 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-trusted-ca-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:32.862800 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.862584 4290 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:32.862800 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.862615 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f8ct\" (UniqueName: \"kubernetes.io/projected/bd5eb143-6167-431f-a09c-5a30a311d67e-kube-api-access-5f8ct\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:32.862800 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:32.862629 4290 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd5eb143-6167-431f-a09c-5a30a311d67e-console-config\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:33.352464 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:33.352434 4290 generic.go:356] "Generic (PLEG): container finished" podID="bd5eb143-6167-431f-a09c-5a30a311d67e" containerID="21bb154822ec080a7815f8bb2c26dd42353a54c217fb1f68cd04045dbe6fe431" exitCode=0 Nov 03 15:10:33.352677 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:33.352486 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774c544669-f6n62" event={"ID":"bd5eb143-6167-431f-a09c-5a30a311d67e","Type":"ContainerDied","Data":"21bb154822ec080a7815f8bb2c26dd42353a54c217fb1f68cd04045dbe6fe431"} Nov 03 15:10:33.352677 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:33.352492 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774c544669-f6n62" Nov 03 15:10:33.352677 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:33.352526 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774c544669-f6n62" event={"ID":"bd5eb143-6167-431f-a09c-5a30a311d67e","Type":"ContainerDied","Data":"7f0acfc84fb5fc598e1bf0246409362df7b196a91e05d6e06333a0e059659cf6"} Nov 03 15:10:33.352677 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:33.352546 4290 scope.go:122] "RemoveContainer" containerID="21bb154822ec080a7815f8bb2c26dd42353a54c217fb1f68cd04045dbe6fe431" Nov 03 15:10:33.361100 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:33.361081 4290 scope.go:122] "RemoveContainer" containerID="21bb154822ec080a7815f8bb2c26dd42353a54c217fb1f68cd04045dbe6fe431" Nov 03 15:10:33.361360 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:10:33.361345 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21bb154822ec080a7815f8bb2c26dd42353a54c217fb1f68cd04045dbe6fe431\": container with ID starting with 21bb154822ec080a7815f8bb2c26dd42353a54c217fb1f68cd04045dbe6fe431 not found: ID does not exist" containerID="21bb154822ec080a7815f8bb2c26dd42353a54c217fb1f68cd04045dbe6fe431" Nov 03 15:10:33.361403 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:33.361368 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21bb154822ec080a7815f8bb2c26dd42353a54c217fb1f68cd04045dbe6fe431"} err="failed to get container status \"21bb154822ec080a7815f8bb2c26dd42353a54c217fb1f68cd04045dbe6fe431\": rpc error: code = NotFound desc = could not find container \"21bb154822ec080a7815f8bb2c26dd42353a54c217fb1f68cd04045dbe6fe431\": container with ID starting with 21bb154822ec080a7815f8bb2c26dd42353a54c217fb1f68cd04045dbe6fe431 not found: ID does not exist" Nov 03 15:10:33.518806 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:33.518770 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-774c544669-f6n62"] Nov 03 15:10:33.573796 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:33.573770 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-774c544669-f6n62"] Nov 03 15:10:34.675625 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:10:34.675595 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd5eb143-6167-431f-a09c-5a30a311d67e" path="/var/lib/kubelet/pods/bd5eb143-6167-431f-a09c-5a30a311d67e/volumes" Nov 03 15:10:38.699684 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:10:38.699647 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:11:08.700749 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:11:08.700648 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:11:38.700690 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:11:38.700650 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:12:08.702646 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:12:08.702562 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:12:14.452764 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.452727 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd"] Nov 03 15:12:14.453230 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.453102 4290 cpu_manager.go:414] "containerMap: removing container" podUID="bd5eb143-6167-431f-a09c-5a30a311d67e" containerName="console" Nov 03 15:12:14.453230 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.453113 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5eb143-6167-431f-a09c-5a30a311d67e" containerName="console" Nov 03 15:12:14.453230 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.453168 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5eb143-6167-431f-a09c-5a30a311d67e" containerName="console" Nov 03 15:12:14.456345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.456329 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" Nov 03 15:12:14.559508 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.559475 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 15:12:14.559694 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.559579 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 15:12:14.559694 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.559594 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-cngdl\"" Nov 03 15:12:14.560840 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.560821 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd"] Nov 03 15:12:14.572115 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.572092 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/190d9f1a-5aa8-4281-9a33-65ed716e2754-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd\" (UID: \"190d9f1a-5aa8-4281-9a33-65ed716e2754\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" Nov 03 15:12:14.572174 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.572147 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8l72\" (UniqueName: \"kubernetes.io/projected/190d9f1a-5aa8-4281-9a33-65ed716e2754-kube-api-access-k8l72\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd\" (UID: \"190d9f1a-5aa8-4281-9a33-65ed716e2754\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" Nov 03 15:12:14.572299 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.572277 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/190d9f1a-5aa8-4281-9a33-65ed716e2754-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd\" (UID: \"190d9f1a-5aa8-4281-9a33-65ed716e2754\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" Nov 03 15:12:14.672931 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.672898 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8l72\" (UniqueName: \"kubernetes.io/projected/190d9f1a-5aa8-4281-9a33-65ed716e2754-kube-api-access-k8l72\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd\" (UID: \"190d9f1a-5aa8-4281-9a33-65ed716e2754\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" Nov 03 15:12:14.673099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.672959 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/190d9f1a-5aa8-4281-9a33-65ed716e2754-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd\" (UID: \"190d9f1a-5aa8-4281-9a33-65ed716e2754\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" Nov 03 15:12:14.673099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.672997 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/190d9f1a-5aa8-4281-9a33-65ed716e2754-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd\" (UID: \"190d9f1a-5aa8-4281-9a33-65ed716e2754\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" Nov 03 15:12:14.673441 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.673409 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/190d9f1a-5aa8-4281-9a33-65ed716e2754-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd\" (UID: \"190d9f1a-5aa8-4281-9a33-65ed716e2754\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" Nov 03 15:12:14.673561 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.673420 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/190d9f1a-5aa8-4281-9a33-65ed716e2754-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd\" (UID: \"190d9f1a-5aa8-4281-9a33-65ed716e2754\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" Nov 03 15:12:14.734855 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.734693 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8l72\" (UniqueName: \"kubernetes.io/projected/190d9f1a-5aa8-4281-9a33-65ed716e2754-kube-api-access-k8l72\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd\" (UID: \"190d9f1a-5aa8-4281-9a33-65ed716e2754\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" Nov 03 15:12:14.771861 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:14.771824 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" Nov 03 15:12:15.146472 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:15.146329 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd"] Nov 03 15:12:15.149372 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:12:15.149339 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190d9f1a_5aa8_4281_9a33_65ed716e2754.slice/crio-48df3f5038d26b57f69587376737503fe6cb9f8517a179889f6e22bfdb27d2e1 WatchSource:0}: Error finding container 48df3f5038d26b57f69587376737503fe6cb9f8517a179889f6e22bfdb27d2e1: Status 404 returned error can't find the container with id 48df3f5038d26b57f69587376737503fe6cb9f8517a179889f6e22bfdb27d2e1 Nov 03 15:12:15.686093 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:15.686060 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9"] Nov 03 15:12:15.690633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:15.690616 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" Nov 03 15:12:15.728427 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:15.728392 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" event={"ID":"190d9f1a-5aa8-4281-9a33-65ed716e2754","Type":"ContainerStarted","Data":"48df3f5038d26b57f69587376737503fe6cb9f8517a179889f6e22bfdb27d2e1"} Nov 03 15:12:15.744575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:15.744547 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-dct2q\"" Nov 03 15:12:15.749622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:15.749586 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9"] Nov 03 15:12:15.781339 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:15.781302 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5deaed3c-667f-4079-a8ce-a697ada69c7f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9\" (UID: \"5deaed3c-667f-4079-a8ce-a697ada69c7f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" Nov 03 15:12:15.781339 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:15.781339 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5deaed3c-667f-4079-a8ce-a697ada69c7f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9\" (UID: \"5deaed3c-667f-4079-a8ce-a697ada69c7f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" Nov 03 15:12:15.781587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:15.781434 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx275\" (UniqueName: \"kubernetes.io/projected/5deaed3c-667f-4079-a8ce-a697ada69c7f-kube-api-access-nx275\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9\" (UID: \"5deaed3c-667f-4079-a8ce-a697ada69c7f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" Nov 03 15:12:15.882332 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:15.882288 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx275\" (UniqueName: \"kubernetes.io/projected/5deaed3c-667f-4079-a8ce-a697ada69c7f-kube-api-access-nx275\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9\" (UID: \"5deaed3c-667f-4079-a8ce-a697ada69c7f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" Nov 03 15:12:15.882509 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:15.882410 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5deaed3c-667f-4079-a8ce-a697ada69c7f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9\" (UID: \"5deaed3c-667f-4079-a8ce-a697ada69c7f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" Nov 03 15:12:15.882509 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:15.882442 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5deaed3c-667f-4079-a8ce-a697ada69c7f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9\" (UID: \"5deaed3c-667f-4079-a8ce-a697ada69c7f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" Nov 03 15:12:15.882863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:15.882847 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5deaed3c-667f-4079-a8ce-a697ada69c7f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9\" (UID: \"5deaed3c-667f-4079-a8ce-a697ada69c7f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" Nov 03 15:12:15.882907 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:15.882867 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5deaed3c-667f-4079-a8ce-a697ada69c7f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9\" (UID: \"5deaed3c-667f-4079-a8ce-a697ada69c7f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" Nov 03 15:12:15.944137 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:15.944054 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx275\" (UniqueName: \"kubernetes.io/projected/5deaed3c-667f-4079-a8ce-a697ada69c7f-kube-api-access-nx275\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9\" (UID: \"5deaed3c-667f-4079-a8ce-a697ada69c7f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" Nov 03 15:12:16.006209 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.006177 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" Nov 03 15:12:16.238918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.238838 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q"] Nov 03 15:12:16.242479 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.242446 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" Nov 03 15:12:16.286897 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.286861 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/778c932f-5f44-476a-b6ca-dd489022e1e6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q\" (UID: \"778c932f-5f44-476a-b6ca-dd489022e1e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" Nov 03 15:12:16.287081 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.286907 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/778c932f-5f44-476a-b6ca-dd489022e1e6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q\" (UID: \"778c932f-5f44-476a-b6ca-dd489022e1e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" Nov 03 15:12:16.287081 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.287051 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kml4m\" (UniqueName: \"kubernetes.io/projected/778c932f-5f44-476a-b6ca-dd489022e1e6-kube-api-access-kml4m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q\" (UID: \"778c932f-5f44-476a-b6ca-dd489022e1e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" Nov 03 15:12:16.294770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.294737 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-8fjmf\"" Nov 03 15:12:16.301560 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.301526 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q"] Nov 03 15:12:16.345999 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.345908 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8"] Nov 03 15:12:16.349908 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.349875 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" Nov 03 15:12:16.387976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.387935 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/778c932f-5f44-476a-b6ca-dd489022e1e6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q\" (UID: \"778c932f-5f44-476a-b6ca-dd489022e1e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" Nov 03 15:12:16.388151 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.387978 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxrtd\" (UniqueName: \"kubernetes.io/projected/9af18a4e-7c82-49f3-9edb-8de433790ea3-kube-api-access-sxrtd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8\" (UID: \"9af18a4e-7c82-49f3-9edb-8de433790ea3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" Nov 03 15:12:16.388151 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.388022 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/778c932f-5f44-476a-b6ca-dd489022e1e6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q\" (UID: \"778c932f-5f44-476a-b6ca-dd489022e1e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" Nov 03 15:12:16.388151 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.388046 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9af18a4e-7c82-49f3-9edb-8de433790ea3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8\" (UID: \"9af18a4e-7c82-49f3-9edb-8de433790ea3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" Nov 03 15:12:16.388283 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.388144 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kml4m\" (UniqueName: \"kubernetes.io/projected/778c932f-5f44-476a-b6ca-dd489022e1e6-kube-api-access-kml4m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q\" (UID: \"778c932f-5f44-476a-b6ca-dd489022e1e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" Nov 03 15:12:16.388283 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.388250 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9af18a4e-7c82-49f3-9edb-8de433790ea3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8\" (UID: \"9af18a4e-7c82-49f3-9edb-8de433790ea3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" Nov 03 15:12:16.388396 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.388376 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/778c932f-5f44-476a-b6ca-dd489022e1e6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q\" (UID: \"778c932f-5f44-476a-b6ca-dd489022e1e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" Nov 03 15:12:16.388441 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.388392 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/778c932f-5f44-476a-b6ca-dd489022e1e6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q\" (UID: \"778c932f-5f44-476a-b6ca-dd489022e1e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" Nov 03 15:12:16.404071 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.404030 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-lg7t2\"" Nov 03 15:12:16.406254 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.406225 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8"] Nov 03 15:12:16.422314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.422281 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9"] Nov 03 15:12:16.426272 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:12:16.426239 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5deaed3c_667f_4079_a8ce_a697ada69c7f.slice/crio-ab325eb4369d58612df2e5b06b7fab78d080d4edb047e5ef4956cca46c38bd91 WatchSource:0}: Error finding container ab325eb4369d58612df2e5b06b7fab78d080d4edb047e5ef4956cca46c38bd91: Status 404 returned error can't find the container with id ab325eb4369d58612df2e5b06b7fab78d080d4edb047e5ef4956cca46c38bd91 Nov 03 15:12:16.439046 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.439019 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r"] Nov 03 15:12:16.442936 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.442914 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" Nov 03 15:12:16.450122 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.450092 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kml4m\" (UniqueName: \"kubernetes.io/projected/778c932f-5f44-476a-b6ca-dd489022e1e6-kube-api-access-kml4m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q\" (UID: \"778c932f-5f44-476a-b6ca-dd489022e1e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" Nov 03 15:12:16.489667 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.489574 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9af18a4e-7c82-49f3-9edb-8de433790ea3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8\" (UID: \"9af18a4e-7c82-49f3-9edb-8de433790ea3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" Nov 03 15:12:16.489827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.489666 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9af18a4e-7c82-49f3-9edb-8de433790ea3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8\" (UID: \"9af18a4e-7c82-49f3-9edb-8de433790ea3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" Nov 03 15:12:16.489827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.489751 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zqf\" (UniqueName: \"kubernetes.io/projected/998236af-9ffc-4b16-9a61-5188945ff1d0-kube-api-access-g9zqf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r\" (UID: \"998236af-9ffc-4b16-9a61-5188945ff1d0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" Nov 03 15:12:16.489827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.489801 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxrtd\" (UniqueName: \"kubernetes.io/projected/9af18a4e-7c82-49f3-9edb-8de433790ea3-kube-api-access-sxrtd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8\" (UID: \"9af18a4e-7c82-49f3-9edb-8de433790ea3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" Nov 03 15:12:16.489969 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.489828 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/998236af-9ffc-4b16-9a61-5188945ff1d0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r\" (UID: \"998236af-9ffc-4b16-9a61-5188945ff1d0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" Nov 03 15:12:16.490078 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.490056 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9af18a4e-7c82-49f3-9edb-8de433790ea3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8\" (UID: \"9af18a4e-7c82-49f3-9edb-8de433790ea3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" Nov 03 15:12:16.490078 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.490069 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9af18a4e-7c82-49f3-9edb-8de433790ea3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8\" (UID: \"9af18a4e-7c82-49f3-9edb-8de433790ea3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" Nov 03 15:12:16.490165 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.490130 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/998236af-9ffc-4b16-9a61-5188945ff1d0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r\" (UID: \"998236af-9ffc-4b16-9a61-5188945ff1d0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" Nov 03 15:12:16.504053 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.504018 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-gbrht\"" Nov 03 15:12:16.509123 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.505758 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r"] Nov 03 15:12:16.538874 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.538832 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv"] Nov 03 15:12:16.543303 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.543271 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv"] Nov 03 15:12:16.543501 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.543413 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" Nov 03 15:12:16.547164 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.547133 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" Nov 03 15:12:16.551419 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.551395 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxrtd\" (UniqueName: \"kubernetes.io/projected/9af18a4e-7c82-49f3-9edb-8de433790ea3-kube-api-access-sxrtd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8\" (UID: \"9af18a4e-7c82-49f3-9edb-8de433790ea3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" Nov 03 15:12:16.561366 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.561344 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" Nov 03 15:12:16.590742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.590684 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkcjk\" (UniqueName: \"kubernetes.io/projected/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-kube-api-access-jkcjk\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv\" (UID: \"29de4a80-6ad0-4d46-bfbe-4f5b013a879f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" Nov 03 15:12:16.591017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.590777 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zqf\" (UniqueName: \"kubernetes.io/projected/998236af-9ffc-4b16-9a61-5188945ff1d0-kube-api-access-g9zqf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r\" (UID: \"998236af-9ffc-4b16-9a61-5188945ff1d0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" Nov 03 15:12:16.591017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.590809 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv\" (UID: \"29de4a80-6ad0-4d46-bfbe-4f5b013a879f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" Nov 03 15:12:16.591017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.590856 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/998236af-9ffc-4b16-9a61-5188945ff1d0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r\" (UID: \"998236af-9ffc-4b16-9a61-5188945ff1d0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" Nov 03 15:12:16.591017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.590879 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/998236af-9ffc-4b16-9a61-5188945ff1d0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r\" (UID: \"998236af-9ffc-4b16-9a61-5188945ff1d0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" Nov 03 15:12:16.591342 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.591032 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c8340b3-582d-4d77-98af-c21b5cbaeeac-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv\" (UID: \"2c8340b3-582d-4d77-98af-c21b5cbaeeac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" Nov 03 15:12:16.591342 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.591144 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c8340b3-582d-4d77-98af-c21b5cbaeeac-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv\" (UID: \"2c8340b3-582d-4d77-98af-c21b5cbaeeac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" Nov 03 15:12:16.591342 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.591176 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmmp7\" (UniqueName: \"kubernetes.io/projected/2c8340b3-582d-4d77-98af-c21b5cbaeeac-kube-api-access-bmmp7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv\" (UID: \"2c8340b3-582d-4d77-98af-c21b5cbaeeac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" Nov 03 15:12:16.591342 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.591194 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv\" (UID: \"29de4a80-6ad0-4d46-bfbe-4f5b013a879f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" Nov 03 15:12:16.591342 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.591256 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/998236af-9ffc-4b16-9a61-5188945ff1d0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r\" (UID: \"998236af-9ffc-4b16-9a61-5188945ff1d0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" Nov 03 15:12:16.591342 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.591308 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/998236af-9ffc-4b16-9a61-5188945ff1d0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r\" (UID: \"998236af-9ffc-4b16-9a61-5188945ff1d0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" Nov 03 15:12:16.601286 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.601254 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv"] Nov 03 15:12:16.603944 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.603910 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv"] Nov 03 15:12:16.604305 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.604287 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-j4kj6\"" Nov 03 15:12:16.604427 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.604405 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-rtkvn\"" Nov 03 15:12:16.652859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.652831 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zqf\" (UniqueName: \"kubernetes.io/projected/998236af-9ffc-4b16-9a61-5188945ff1d0-kube-api-access-g9zqf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r\" (UID: \"998236af-9ffc-4b16-9a61-5188945ff1d0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" Nov 03 15:12:16.668302 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.668269 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" Nov 03 15:12:16.693389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.692315 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c8340b3-582d-4d77-98af-c21b5cbaeeac-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv\" (UID: \"2c8340b3-582d-4d77-98af-c21b5cbaeeac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" Nov 03 15:12:16.693389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.692400 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c8340b3-582d-4d77-98af-c21b5cbaeeac-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv\" (UID: \"2c8340b3-582d-4d77-98af-c21b5cbaeeac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" Nov 03 15:12:16.693389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.692439 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmmp7\" (UniqueName: \"kubernetes.io/projected/2c8340b3-582d-4d77-98af-c21b5cbaeeac-kube-api-access-bmmp7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv\" (UID: \"2c8340b3-582d-4d77-98af-c21b5cbaeeac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" Nov 03 15:12:16.693389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.692468 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv\" (UID: \"29de4a80-6ad0-4d46-bfbe-4f5b013a879f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" Nov 03 15:12:16.693389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.692499 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkcjk\" (UniqueName: \"kubernetes.io/projected/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-kube-api-access-jkcjk\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv\" (UID: \"29de4a80-6ad0-4d46-bfbe-4f5b013a879f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" Nov 03 15:12:16.693389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.692540 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv\" (UID: \"29de4a80-6ad0-4d46-bfbe-4f5b013a879f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" Nov 03 15:12:16.693389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.692799 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c8340b3-582d-4d77-98af-c21b5cbaeeac-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv\" (UID: \"2c8340b3-582d-4d77-98af-c21b5cbaeeac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" Nov 03 15:12:16.693389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.692928 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv\" (UID: \"29de4a80-6ad0-4d46-bfbe-4f5b013a879f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" Nov 03 15:12:16.693389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.693143 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv\" (UID: \"29de4a80-6ad0-4d46-bfbe-4f5b013a879f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" Nov 03 15:12:16.693389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.693339 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c8340b3-582d-4d77-98af-c21b5cbaeeac-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv\" (UID: \"2c8340b3-582d-4d77-98af-c21b5cbaeeac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" Nov 03 15:12:16.741882 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.739651 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" event={"ID":"5deaed3c-667f-4079-a8ce-a697ada69c7f","Type":"ContainerStarted","Data":"ab325eb4369d58612df2e5b06b7fab78d080d4edb047e5ef4956cca46c38bd91"} Nov 03 15:12:16.755596 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.755539 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmmp7\" (UniqueName: \"kubernetes.io/projected/2c8340b3-582d-4d77-98af-c21b5cbaeeac-kube-api-access-bmmp7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv\" (UID: \"2c8340b3-582d-4d77-98af-c21b5cbaeeac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" Nov 03 15:12:16.756218 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.756188 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkcjk\" (UniqueName: \"kubernetes.io/projected/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-kube-api-access-jkcjk\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv\" (UID: \"29de4a80-6ad0-4d46-bfbe-4f5b013a879f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" Nov 03 15:12:16.775947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.775910 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" Nov 03 15:12:16.866196 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.866161 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" Nov 03 15:12:16.873408 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.873374 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" Nov 03 15:12:16.950448 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:16.950340 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q"] Nov 03 15:12:17.054922 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:17.054882 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8"] Nov 03 15:12:17.057777 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:12:17.057744 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af18a4e_7c82_49f3_9edb_8de433790ea3.slice/crio-32af132b174262d288fea8e062ef7d2fd79426cb3c725838942ee39b3e3a5ca2 WatchSource:0}: Error finding container 32af132b174262d288fea8e062ef7d2fd79426cb3c725838942ee39b3e3a5ca2: Status 404 returned error can't find the container with id 32af132b174262d288fea8e062ef7d2fd79426cb3c725838942ee39b3e3a5ca2 Nov 03 15:12:17.181243 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:17.181215 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r"] Nov 03 15:12:17.184051 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:12:17.184016 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998236af_9ffc_4b16_9a61_5188945ff1d0.slice/crio-294cda7c18d8aa8cccba6f7bbec16f70b4ce410dee51cf5cc647ee95b40d0292 WatchSource:0}: Error finding container 294cda7c18d8aa8cccba6f7bbec16f70b4ce410dee51cf5cc647ee95b40d0292: Status 404 returned error can't find the container with id 294cda7c18d8aa8cccba6f7bbec16f70b4ce410dee51cf5cc647ee95b40d0292 Nov 03 15:12:17.281937 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:17.281831 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv"] Nov 03 15:12:17.284920 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:17.284362 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv"] Nov 03 15:12:17.286829 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:12:17.286800 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29de4a80_6ad0_4d46_bfbe_4f5b013a879f.slice/crio-1af4fe8da6a7085d95de6fe25fd33d0abfffba33fcfda17d1ef3a813d0f0c724 WatchSource:0}: Error finding container 1af4fe8da6a7085d95de6fe25fd33d0abfffba33fcfda17d1ef3a813d0f0c724: Status 404 returned error can't find the container with id 1af4fe8da6a7085d95de6fe25fd33d0abfffba33fcfda17d1ef3a813d0f0c724 Nov 03 15:12:17.289033 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:12:17.289012 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8340b3_582d_4d77_98af_c21b5cbaeeac.slice/crio-bf56b7dc80c5306649c66825dd32add27d7c693a13dca572853a8b4204d94a23 WatchSource:0}: Error finding container bf56b7dc80c5306649c66825dd32add27d7c693a13dca572853a8b4204d94a23: Status 404 returned error can't find the container with id bf56b7dc80c5306649c66825dd32add27d7c693a13dca572853a8b4204d94a23 Nov 03 15:12:17.746785 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:17.746690 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" event={"ID":"29de4a80-6ad0-4d46-bfbe-4f5b013a879f","Type":"ContainerStarted","Data":"1af4fe8da6a7085d95de6fe25fd33d0abfffba33fcfda17d1ef3a813d0f0c724"} Nov 03 15:12:17.748047 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:17.748014 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" event={"ID":"2c8340b3-582d-4d77-98af-c21b5cbaeeac","Type":"ContainerStarted","Data":"bf56b7dc80c5306649c66825dd32add27d7c693a13dca572853a8b4204d94a23"} Nov 03 15:12:17.749527 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:17.749499 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" event={"ID":"998236af-9ffc-4b16-9a61-5188945ff1d0","Type":"ContainerStarted","Data":"294cda7c18d8aa8cccba6f7bbec16f70b4ce410dee51cf5cc647ee95b40d0292"} Nov 03 15:12:17.751139 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:17.751105 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" event={"ID":"9af18a4e-7c82-49f3-9edb-8de433790ea3","Type":"ContainerStarted","Data":"32af132b174262d288fea8e062ef7d2fd79426cb3c725838942ee39b3e3a5ca2"} Nov 03 15:12:17.753009 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:17.752985 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" event={"ID":"778c932f-5f44-476a-b6ca-dd489022e1e6","Type":"ContainerStarted","Data":"fdce56443558fcd92019ce55d2913f11c48751b209a5b519dfbc83b4b64f2d95"} Nov 03 15:12:19.827188 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:19.827149 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc"] Nov 03 15:12:19.832280 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:19.832251 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" Nov 03 15:12:19.890825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:19.890788 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc"] Nov 03 15:12:19.929083 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:19.929046 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d7d6100-6217-4649-a0dc-67879e4b0b79-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc\" (UID: \"0d7d6100-6217-4649-a0dc-67879e4b0b79\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" Nov 03 15:12:19.929263 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:19.929231 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z2cp\" (UniqueName: \"kubernetes.io/projected/0d7d6100-6217-4649-a0dc-67879e4b0b79-kube-api-access-9z2cp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc\" (UID: \"0d7d6100-6217-4649-a0dc-67879e4b0b79\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" Nov 03 15:12:19.929336 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:19.929297 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d7d6100-6217-4649-a0dc-67879e4b0b79-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc\" (UID: \"0d7d6100-6217-4649-a0dc-67879e4b0b79\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" Nov 03 15:12:20.031785 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:20.030433 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d7d6100-6217-4649-a0dc-67879e4b0b79-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc\" (UID: \"0d7d6100-6217-4649-a0dc-67879e4b0b79\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" Nov 03 15:12:20.031785 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:20.030489 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d7d6100-6217-4649-a0dc-67879e4b0b79-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc\" (UID: \"0d7d6100-6217-4649-a0dc-67879e4b0b79\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" Nov 03 15:12:20.031785 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:20.030568 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z2cp\" (UniqueName: \"kubernetes.io/projected/0d7d6100-6217-4649-a0dc-67879e4b0b79-kube-api-access-9z2cp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc\" (UID: \"0d7d6100-6217-4649-a0dc-67879e4b0b79\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" Nov 03 15:12:20.031785 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:20.031358 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d7d6100-6217-4649-a0dc-67879e4b0b79-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc\" (UID: \"0d7d6100-6217-4649-a0dc-67879e4b0b79\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" Nov 03 15:12:20.031785 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:20.031731 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d7d6100-6217-4649-a0dc-67879e4b0b79-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc\" (UID: \"0d7d6100-6217-4649-a0dc-67879e4b0b79\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" Nov 03 15:12:20.102493 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:20.102411 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z2cp\" (UniqueName: \"kubernetes.io/projected/0d7d6100-6217-4649-a0dc-67879e4b0b79-kube-api-access-9z2cp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc\" (UID: \"0d7d6100-6217-4649-a0dc-67879e4b0b79\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" Nov 03 15:12:20.156019 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:20.155693 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" Nov 03 15:12:20.569891 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:20.569857 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc"] Nov 03 15:12:20.581622 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:12:20.581580 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7d6100_6217_4649_a0dc_67879e4b0b79.slice/crio-086a0ca300b0f4a5682fe3e81f731e8faff7c05b43887bdc543502ddb3f2b494 WatchSource:0}: Error finding container 086a0ca300b0f4a5682fe3e81f731e8faff7c05b43887bdc543502ddb3f2b494: Status 404 returned error can't find the container with id 086a0ca300b0f4a5682fe3e81f731e8faff7c05b43887bdc543502ddb3f2b494 Nov 03 15:12:20.787261 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:20.787184 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" event={"ID":"0d7d6100-6217-4649-a0dc-67879e4b0b79","Type":"ContainerStarted","Data":"086a0ca300b0f4a5682fe3e81f731e8faff7c05b43887bdc543502ddb3f2b494"} Nov 03 15:12:21.618258 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.618222 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2"] Nov 03 15:12:21.622682 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.621256 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.679066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.678744 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 15:12:21.679442 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.679409 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-n82lg\"" Nov 03 15:12:21.695662 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.695574 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 15:12:21.728306 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.728243 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 15:12:21.730431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.730264 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2"] Nov 03 15:12:21.754107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.752457 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871baa2b-806e-4033-a553-277c509a3c14-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.754107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.752577 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871baa2b-806e-4033-a553-277c509a3c14-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.754107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.752614 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871baa2b-806e-4033-a553-277c509a3c14-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.754107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.752713 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871baa2b-806e-4033-a553-277c509a3c14-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.754107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.752769 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baa2b-806e-4033-a553-277c509a3c14-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.754107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.752802 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871baa2b-806e-4033-a553-277c509a3c14-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.754107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.752828 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8nnn\" (UniqueName: \"kubernetes.io/projected/871baa2b-806e-4033-a553-277c509a3c14-kube-api-access-d8nnn\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.754107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.752857 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871baa2b-806e-4033-a553-277c509a3c14-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.754107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.752889 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baa2b-806e-4033-a553-277c509a3c14-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.853833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.853787 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871baa2b-806e-4033-a553-277c509a3c14-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.854017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.853853 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871baa2b-806e-4033-a553-277c509a3c14-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.854017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.853897 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871baa2b-806e-4033-a553-277c509a3c14-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.854017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.853966 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871baa2b-806e-4033-a553-277c509a3c14-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.854017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.854011 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baa2b-806e-4033-a553-277c509a3c14-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.854234 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.854050 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871baa2b-806e-4033-a553-277c509a3c14-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.854234 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.854080 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8nnn\" (UniqueName: \"kubernetes.io/projected/871baa2b-806e-4033-a553-277c509a3c14-kube-api-access-d8nnn\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.854234 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.854113 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871baa2b-806e-4033-a553-277c509a3c14-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.854234 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.854148 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baa2b-806e-4033-a553-277c509a3c14-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.854801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.854773 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baa2b-806e-4033-a553-277c509a3c14-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.855484 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.855460 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871baa2b-806e-4033-a553-277c509a3c14-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.855787 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.855765 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871baa2b-806e-4033-a553-277c509a3c14-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.856041 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.856020 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871baa2b-806e-4033-a553-277c509a3c14-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.856733 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.856693 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baa2b-806e-4033-a553-277c509a3c14-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.864107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.864040 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871baa2b-806e-4033-a553-277c509a3c14-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.879092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.878987 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871baa2b-806e-4033-a553-277c509a3c14-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.929289 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.929220 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871baa2b-806e-4033-a553-277c509a3c14-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.932943 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.932833 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8nnn\" (UniqueName: \"kubernetes.io/projected/871baa2b-806e-4033-a553-277c509a3c14-kube-api-access-d8nnn\") pod \"maas-default-gateway-openshift-default-596dc9545f-2sbg2\" (UID: \"871baa2b-806e-4033-a553-277c509a3c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.944101 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.943580 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z"] Nov 03 15:12:21.944101 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.943858 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:21.947487 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:21.947430 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.001153 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.001100 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-vxbxn\"" Nov 03 15:12:22.011638 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.011604 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z"] Nov 03 15:12:22.057124 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.057081 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5daa2f8-b704-445c-b085-ad6fb0b52063-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.057313 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.057145 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5daa2f8-b704-445c-b085-ad6fb0b52063-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.058348 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.058318 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5daa2f8-b704-445c-b085-ad6fb0b52063-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.059918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.059370 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5daa2f8-b704-445c-b085-ad6fb0b52063-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.059918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.059446 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5daa2f8-b704-445c-b085-ad6fb0b52063-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.059918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.059485 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5daa2f8-b704-445c-b085-ad6fb0b52063-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.059918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.059510 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56hh5\" (UniqueName: \"kubernetes.io/projected/b5daa2f8-b704-445c-b085-ad6fb0b52063-kube-api-access-56hh5\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.059918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.059554 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5daa2f8-b704-445c-b085-ad6fb0b52063-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.059918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.059581 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5daa2f8-b704-445c-b085-ad6fb0b52063-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.161123 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.161015 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5daa2f8-b704-445c-b085-ad6fb0b52063-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.161123 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.161075 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56hh5\" (UniqueName: \"kubernetes.io/projected/b5daa2f8-b704-445c-b085-ad6fb0b52063-kube-api-access-56hh5\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.161123 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.161120 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5daa2f8-b704-445c-b085-ad6fb0b52063-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.161405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.161148 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5daa2f8-b704-445c-b085-ad6fb0b52063-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.161405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.161229 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5daa2f8-b704-445c-b085-ad6fb0b52063-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.161405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.161265 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5daa2f8-b704-445c-b085-ad6fb0b52063-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.161405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.161295 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5daa2f8-b704-445c-b085-ad6fb0b52063-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.161405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.161341 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5daa2f8-b704-445c-b085-ad6fb0b52063-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.161405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.161394 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5daa2f8-b704-445c-b085-ad6fb0b52063-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.162354 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.162295 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5daa2f8-b704-445c-b085-ad6fb0b52063-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.162592 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.162567 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5daa2f8-b704-445c-b085-ad6fb0b52063-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.163058 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.162808 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5daa2f8-b704-445c-b085-ad6fb0b52063-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.163058 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.162906 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5daa2f8-b704-445c-b085-ad6fb0b52063-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.163225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.163149 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5daa2f8-b704-445c-b085-ad6fb0b52063-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.166963 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.166936 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5daa2f8-b704-445c-b085-ad6fb0b52063-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.167464 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.167419 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5daa2f8-b704-445c-b085-ad6fb0b52063-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.227665 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.227562 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56hh5\" (UniqueName: \"kubernetes.io/projected/b5daa2f8-b704-445c-b085-ad6fb0b52063-kube-api-access-56hh5\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.246491 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.246408 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5daa2f8-b704-445c-b085-ad6fb0b52063-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z\" (UID: \"b5daa2f8-b704-445c-b085-ad6fb0b52063\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.266558 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.266168 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:22.460748 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.460633 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2"] Nov 03 15:12:22.471872 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:12:22.471835 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871baa2b_806e_4033_a553_277c509a3c14.slice/crio-5814f529b4b8317705b961cb9a7f6e0a4f56b1f70ae693da49287868226c6fc2 WatchSource:0}: Error finding container 5814f529b4b8317705b961cb9a7f6e0a4f56b1f70ae693da49287868226c6fc2: Status 404 returned error can't find the container with id 5814f529b4b8317705b961cb9a7f6e0a4f56b1f70ae693da49287868226c6fc2 Nov 03 15:12:22.699909 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.698398 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k"] Nov 03 15:12:22.705484 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.703951 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" Nov 03 15:12:22.708611 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.705773 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z"] Nov 03 15:12:22.714386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.714362 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2"] Nov 03 15:12:22.725142 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.725118 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" Nov 03 15:12:22.739558 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.739511 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k"] Nov 03 15:12:22.755435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.755407 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z"] Nov 03 15:12:22.759410 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.759388 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" Nov 03 15:12:22.767723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.767673 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt682\" (UniqueName: \"kubernetes.io/projected/f351bf29-892f-4c6f-9c1a-b284adb73e58-kube-api-access-vt682\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k\" (UID: \"f351bf29-892f-4c6f-9c1a-b284adb73e58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" Nov 03 15:12:22.768202 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.768072 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrx7d\" (UniqueName: \"kubernetes.io/projected/a529a542-a350-4e78-a685-bb48986be4e1-kube-api-access-vrx7d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2\" (UID: \"a529a542-a350-4e78-a685-bb48986be4e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" Nov 03 15:12:22.768539 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.768518 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a529a542-a350-4e78-a685-bb48986be4e1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2\" (UID: \"a529a542-a350-4e78-a685-bb48986be4e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" Nov 03 15:12:22.768737 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.768721 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f351bf29-892f-4c6f-9c1a-b284adb73e58-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k\" (UID: \"f351bf29-892f-4c6f-9c1a-b284adb73e58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" Nov 03 15:12:22.768876 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.768862 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f351bf29-892f-4c6f-9c1a-b284adb73e58-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k\" (UID: \"f351bf29-892f-4c6f-9c1a-b284adb73e58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" Nov 03 15:12:22.769057 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.769041 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a529a542-a350-4e78-a685-bb48986be4e1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2\" (UID: \"a529a542-a350-4e78-a685-bb48986be4e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" Nov 03 15:12:22.786395 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.786364 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8"] Nov 03 15:12:22.790501 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.790474 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" Nov 03 15:12:22.806658 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.806634 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2"] Nov 03 15:12:22.827724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.826813 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" event={"ID":"b5daa2f8-b704-445c-b085-ad6fb0b52063","Type":"ContainerStarted","Data":"e266e06dba0f95ec99861b87b15b6bbe059cadc398fcd7e6c6f63e1bcc40587b"} Nov 03 15:12:22.833874 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.833845 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z"] Nov 03 15:12:22.834476 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.834445 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" event={"ID":"871baa2b-806e-4033-a553-277c509a3c14","Type":"ContainerStarted","Data":"5814f529b4b8317705b961cb9a7f6e0a4f56b1f70ae693da49287868226c6fc2"} Nov 03 15:12:22.850246 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.850223 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8"] Nov 03 15:12:22.871623 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.871583 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f351bf29-892f-4c6f-9c1a-b284adb73e58-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k\" (UID: \"f351bf29-892f-4c6f-9c1a-b284adb73e58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" Nov 03 15:12:22.871792 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.871631 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc6a18ed-d1fe-436e-afad-2acfec2954cb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z\" (UID: \"bc6a18ed-d1fe-436e-afad-2acfec2954cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" Nov 03 15:12:22.871792 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.871670 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f351bf29-892f-4c6f-9c1a-b284adb73e58-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k\" (UID: \"f351bf29-892f-4c6f-9c1a-b284adb73e58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" Nov 03 15:12:22.871792 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.871731 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a529a542-a350-4e78-a685-bb48986be4e1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2\" (UID: \"a529a542-a350-4e78-a685-bb48986be4e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" Nov 03 15:12:22.871792 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.871759 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03a0fb03-c422-430a-aa4b-6226262bcb73-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8\" (UID: \"03a0fb03-c422-430a-aa4b-6226262bcb73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" Nov 03 15:12:22.871958 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.871798 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt682\" (UniqueName: \"kubernetes.io/projected/f351bf29-892f-4c6f-9c1a-b284adb73e58-kube-api-access-vt682\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k\" (UID: \"f351bf29-892f-4c6f-9c1a-b284adb73e58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" Nov 03 15:12:22.871958 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.871835 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a529a542-a350-4e78-a685-bb48986be4e1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2\" (UID: \"a529a542-a350-4e78-a685-bb48986be4e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" Nov 03 15:12:22.871958 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.871892 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46p8b\" (UniqueName: \"kubernetes.io/projected/03a0fb03-c422-430a-aa4b-6226262bcb73-kube-api-access-46p8b\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8\" (UID: \"03a0fb03-c422-430a-aa4b-6226262bcb73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" Nov 03 15:12:22.871958 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.871927 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc6a18ed-d1fe-436e-afad-2acfec2954cb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z\" (UID: \"bc6a18ed-d1fe-436e-afad-2acfec2954cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" Nov 03 15:12:22.872083 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.871956 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03a0fb03-c422-430a-aa4b-6226262bcb73-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8\" (UID: \"03a0fb03-c422-430a-aa4b-6226262bcb73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" Nov 03 15:12:22.872083 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.871998 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrx7d\" (UniqueName: \"kubernetes.io/projected/a529a542-a350-4e78-a685-bb48986be4e1-kube-api-access-vrx7d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2\" (UID: \"a529a542-a350-4e78-a685-bb48986be4e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" Nov 03 15:12:22.872083 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.872024 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmbgw\" (UniqueName: \"kubernetes.io/projected/bc6a18ed-d1fe-436e-afad-2acfec2954cb-kube-api-access-gmbgw\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z\" (UID: \"bc6a18ed-d1fe-436e-afad-2acfec2954cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" Nov 03 15:12:22.872487 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.872465 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f351bf29-892f-4c6f-9c1a-b284adb73e58-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k\" (UID: \"f351bf29-892f-4c6f-9c1a-b284adb73e58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" Nov 03 15:12:22.872769 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.872751 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f351bf29-892f-4c6f-9c1a-b284adb73e58-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k\" (UID: \"f351bf29-892f-4c6f-9c1a-b284adb73e58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" Nov 03 15:12:22.873053 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.873033 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a529a542-a350-4e78-a685-bb48986be4e1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2\" (UID: \"a529a542-a350-4e78-a685-bb48986be4e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" Nov 03 15:12:22.873631 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.873601 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a529a542-a350-4e78-a685-bb48986be4e1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2\" (UID: \"a529a542-a350-4e78-a685-bb48986be4e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" Nov 03 15:12:22.938438 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.938402 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt682\" (UniqueName: \"kubernetes.io/projected/f351bf29-892f-4c6f-9c1a-b284adb73e58-kube-api-access-vt682\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k\" (UID: \"f351bf29-892f-4c6f-9c1a-b284adb73e58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" Nov 03 15:12:22.942581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.942551 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrx7d\" (UniqueName: \"kubernetes.io/projected/a529a542-a350-4e78-a685-bb48986be4e1-kube-api-access-vrx7d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2\" (UID: \"a529a542-a350-4e78-a685-bb48986be4e1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" Nov 03 15:12:22.973470 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.973344 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46p8b\" (UniqueName: \"kubernetes.io/projected/03a0fb03-c422-430a-aa4b-6226262bcb73-kube-api-access-46p8b\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8\" (UID: \"03a0fb03-c422-430a-aa4b-6226262bcb73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" Nov 03 15:12:22.973470 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.973399 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc6a18ed-d1fe-436e-afad-2acfec2954cb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z\" (UID: \"bc6a18ed-d1fe-436e-afad-2acfec2954cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" Nov 03 15:12:22.973470 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.973426 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03a0fb03-c422-430a-aa4b-6226262bcb73-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8\" (UID: \"03a0fb03-c422-430a-aa4b-6226262bcb73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" Nov 03 15:12:22.973470 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.973460 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbgw\" (UniqueName: \"kubernetes.io/projected/bc6a18ed-d1fe-436e-afad-2acfec2954cb-kube-api-access-gmbgw\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z\" (UID: \"bc6a18ed-d1fe-436e-afad-2acfec2954cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" Nov 03 15:12:22.973822 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.973521 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc6a18ed-d1fe-436e-afad-2acfec2954cb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z\" (UID: \"bc6a18ed-d1fe-436e-afad-2acfec2954cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" Nov 03 15:12:22.973822 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.973569 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03a0fb03-c422-430a-aa4b-6226262bcb73-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8\" (UID: \"03a0fb03-c422-430a-aa4b-6226262bcb73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" Nov 03 15:12:22.974582 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.974040 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03a0fb03-c422-430a-aa4b-6226262bcb73-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8\" (UID: \"03a0fb03-c422-430a-aa4b-6226262bcb73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" Nov 03 15:12:22.975490 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.974686 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc6a18ed-d1fe-436e-afad-2acfec2954cb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z\" (UID: \"bc6a18ed-d1fe-436e-afad-2acfec2954cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" Nov 03 15:12:22.975599 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.975179 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03a0fb03-c422-430a-aa4b-6226262bcb73-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8\" (UID: \"03a0fb03-c422-430a-aa4b-6226262bcb73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" Nov 03 15:12:22.975599 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:22.975392 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc6a18ed-d1fe-436e-afad-2acfec2954cb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z\" (UID: \"bc6a18ed-d1fe-436e-afad-2acfec2954cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" Nov 03 15:12:23.030121 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:23.030085 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" Nov 03 15:12:23.040148 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:23.040089 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46p8b\" (UniqueName: \"kubernetes.io/projected/03a0fb03-c422-430a-aa4b-6226262bcb73-kube-api-access-46p8b\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8\" (UID: \"03a0fb03-c422-430a-aa4b-6226262bcb73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" Nov 03 15:12:23.047067 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:23.047041 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" Nov 03 15:12:23.048281 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:23.048258 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmbgw\" (UniqueName: \"kubernetes.io/projected/bc6a18ed-d1fe-436e-afad-2acfec2954cb-kube-api-access-gmbgw\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z\" (UID: \"bc6a18ed-d1fe-436e-afad-2acfec2954cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" Nov 03 15:12:23.081637 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:23.081597 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" Nov 03 15:12:23.135411 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:23.135194 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" Nov 03 15:12:23.481726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:23.481485 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k"] Nov 03 15:12:23.485006 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:23.484983 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2"] Nov 03 15:12:23.490464 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:12:23.490425 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda529a542_a350_4e78_a685_bb48986be4e1.slice/crio-3f975713e5d9e6e59e36ff1e66889928a2122bf6de0e99dd3bee364fb5fddc2a WatchSource:0}: Error finding container 3f975713e5d9e6e59e36ff1e66889928a2122bf6de0e99dd3bee364fb5fddc2a: Status 404 returned error can't find the container with id 3f975713e5d9e6e59e36ff1e66889928a2122bf6de0e99dd3bee364fb5fddc2a Nov 03 15:12:23.543393 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:23.543313 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z"] Nov 03 15:12:23.623945 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:23.623900 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8"] Nov 03 15:12:23.847343 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:23.847237 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" event={"ID":"03a0fb03-c422-430a-aa4b-6226262bcb73","Type":"ContainerStarted","Data":"99490aebf342de636c678c416c3bca566777346ca2003c19edd036b029e3150f"} Nov 03 15:12:23.854335 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:23.854302 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" event={"ID":"f351bf29-892f-4c6f-9c1a-b284adb73e58","Type":"ContainerStarted","Data":"3960ab939e4208fead97157c7de665aa9e742a0ac7e83a1018b4c45438ee20fb"} Nov 03 15:12:23.856866 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:23.856806 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" event={"ID":"bc6a18ed-d1fe-436e-afad-2acfec2954cb","Type":"ContainerStarted","Data":"fc7cba88187722f83380dadda034af391db367c92524954b04efee1879d37077"} Nov 03 15:12:23.858452 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:23.858430 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" event={"ID":"a529a542-a350-4e78-a685-bb48986be4e1","Type":"ContainerStarted","Data":"3f975713e5d9e6e59e36ff1e66889928a2122bf6de0e99dd3bee364fb5fddc2a"} Nov 03 15:12:35.010233 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:35.008581 4290 generic.go:356] "Generic (PLEG): container finished" podID="190d9f1a-5aa8-4281-9a33-65ed716e2754" containerID="9229428a0a9d4e66f63258273112615cab8247cae21dd32544c4629717a42a72" exitCode=0 Nov 03 15:12:35.010233 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:35.008634 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" event={"ID":"190d9f1a-5aa8-4281-9a33-65ed716e2754","Type":"ContainerDied","Data":"9229428a0a9d4e66f63258273112615cab8247cae21dd32544c4629717a42a72"} Nov 03 15:12:36.340164 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:36.340113 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:12:36.340667 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:36.340203 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:12:36.340667 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:36.340255 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:12:36.367539 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:36.366970 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:12:36.367539 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:36.367077 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:12:36.367539 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:36.367128 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:12:37.033987 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:37.033939 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" event={"ID":"b5daa2f8-b704-445c-b085-ad6fb0b52063","Type":"ContainerStarted","Data":"ae134cbe4321220e7b953a81d380e0066efa78fb34f52792b3b6e978a1192e66"} Nov 03 15:12:37.042329 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:37.041624 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" event={"ID":"871baa2b-806e-4033-a553-277c509a3c14","Type":"ContainerStarted","Data":"456abefd97b059f39907e4ac211626e46a1bd334aa9215410394579a01825024"} Nov 03 15:12:37.048294 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:37.046863 4290 generic.go:356] "Generic (PLEG): container finished" podID="0d7d6100-6217-4649-a0dc-67879e4b0b79" containerID="1019281fa3c81cac8c943bd32d055c0e8f4d6a2e4cd4ee687049e4953f398b8c" exitCode=0 Nov 03 15:12:37.048294 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:37.046936 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" event={"ID":"0d7d6100-6217-4649-a0dc-67879e4b0b79","Type":"ContainerDied","Data":"1019281fa3c81cac8c943bd32d055c0e8f4d6a2e4cd4ee687049e4953f398b8c"} Nov 03 15:12:37.206061 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:37.205995 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" podStartSLOduration=2.561130618 podStartE2EDuration="16.205972873s" totalImagesPullingTime="13.644842255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:12:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:12:37.155099672 +0000 UTC m=+264.586057479" watchObservedRunningTime="2026-11-03 15:12:37.205972873 +0000 UTC m=+264.636930682" Nov 03 15:12:37.269136 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:37.266841 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:37.269136 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:37.267151 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:37.274949 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:37.274895 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:37.281090 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:37.280211 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" podStartSLOduration=2.4153166 podStartE2EDuration="16.280190379s" totalImagesPullingTime="13.864873779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:12:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:12:37.277916637 +0000 UTC m=+264.708874443" watchObservedRunningTime="2026-11-03 15:12:37.280190379 +0000 UTC m=+264.711148188" Nov 03 15:12:37.945465 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:37.945171 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:37.952561 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:37.952305 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:38.075952 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:38.075921 4290 generic.go:356] "Generic (PLEG): container finished" podID="190d9f1a-5aa8-4281-9a33-65ed716e2754" containerID="64f741add7c6a3908f0d77733a2c84fc0fd9035cd503bfc9690a8c118acf98f6" exitCode=0 Nov 03 15:12:38.080603 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:38.080562 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" event={"ID":"190d9f1a-5aa8-4281-9a33-65ed716e2754","Type":"ContainerDied","Data":"64f741add7c6a3908f0d77733a2c84fc0fd9035cd503bfc9690a8c118acf98f6"} Nov 03 15:12:38.080735 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:38.080630 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:38.081186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:38.081168 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z" Nov 03 15:12:38.083627 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:38.083563 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:12:38.345036 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:12:38.344990 4290 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7d6100_6217_4649_a0dc_67879e4b0b79.slice/crio-conmon-84cd8b1d9aac2f34e677030b956da82fe0fc732ae4f0114d5a1f9d5535b289f5.scope\": RecentStats: unable to find data in memory cache]" Nov 03 15:12:38.346084 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:12:38.346023 4290 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7d6100_6217_4649_a0dc_67879e4b0b79.slice/crio-conmon-84cd8b1d9aac2f34e677030b956da82fe0fc732ae4f0114d5a1f9d5535b289f5.scope\": RecentStats: unable to find data in memory cache]" Nov 03 15:12:38.715811 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:12:38.715729 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:12:39.084836 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:39.084080 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" event={"ID":"03a0fb03-c422-430a-aa4b-6226262bcb73","Type":"ContainerStarted","Data":"127b5048f2877d6ebc23e3f2246dce2df2e6ad8dc344a505271bfdb6f91c7424"} Nov 03 15:12:39.087584 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:39.087547 4290 generic.go:356] "Generic (PLEG): container finished" podID="f351bf29-892f-4c6f-9c1a-b284adb73e58" containerID="3d52e80628ab86a818f0db46243ab09cf7e945c98f4cf9404a3d8d20fa730756" exitCode=0 Nov 03 15:12:39.087939 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:39.087908 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" event={"ID":"f351bf29-892f-4c6f-9c1a-b284adb73e58","Type":"ContainerDied","Data":"3d52e80628ab86a818f0db46243ab09cf7e945c98f4cf9404a3d8d20fa730756"} Nov 03 15:12:39.090107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:39.090081 4290 generic.go:356] "Generic (PLEG): container finished" podID="bc6a18ed-d1fe-436e-afad-2acfec2954cb" containerID="e439567305ad682c85969ca737a5e33cce69e38c303923a5493e591aa987b617" exitCode=0 Nov 03 15:12:39.097502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:39.090167 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" event={"ID":"bc6a18ed-d1fe-436e-afad-2acfec2954cb","Type":"ContainerDied","Data":"e439567305ad682c85969ca737a5e33cce69e38c303923a5493e591aa987b617"} Nov 03 15:12:39.097502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:39.091921 4290 generic.go:356] "Generic (PLEG): container finished" podID="a529a542-a350-4e78-a685-bb48986be4e1" containerID="6ddb798339abf495c54230d247d7af1d2c2e0da015262ddf0fd86e8e3c1a0c7b" exitCode=0 Nov 03 15:12:39.097502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:39.092027 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" event={"ID":"a529a542-a350-4e78-a685-bb48986be4e1","Type":"ContainerDied","Data":"6ddb798339abf495c54230d247d7af1d2c2e0da015262ddf0fd86e8e3c1a0c7b"} Nov 03 15:12:39.097502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:39.094830 4290 generic.go:356] "Generic (PLEG): container finished" podID="0d7d6100-6217-4649-a0dc-67879e4b0b79" containerID="84cd8b1d9aac2f34e677030b956da82fe0fc732ae4f0114d5a1f9d5535b289f5" exitCode=0 Nov 03 15:12:39.097502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:39.094924 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" event={"ID":"0d7d6100-6217-4649-a0dc-67879e4b0b79","Type":"ContainerDied","Data":"84cd8b1d9aac2f34e677030b956da82fe0fc732ae4f0114d5a1f9d5535b289f5"} Nov 03 15:12:40.103731 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:40.103124 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" event={"ID":"5deaed3c-667f-4079-a8ce-a697ada69c7f","Type":"ContainerStarted","Data":"902c620afb802ab59b6ddf5a0cf7245cc85b86b320e24029f57399cc8b1a6017"} Nov 03 15:12:40.105393 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:40.105361 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" event={"ID":"29de4a80-6ad0-4d46-bfbe-4f5b013a879f","Type":"ContainerStarted","Data":"0c360a1f4a4b50c8c52cf3f478b84836c4130555c60feb752373e7295e2cbe50"} Nov 03 15:12:40.107327 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:40.107297 4290 generic.go:356] "Generic (PLEG): container finished" podID="2c8340b3-582d-4d77-98af-c21b5cbaeeac" containerID="84c6f6f7663514af84eeb96927c8cd471205967d5b0e423193fca3306842e012" exitCode=0 Nov 03 15:12:40.107444 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:40.107351 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" event={"ID":"2c8340b3-582d-4d77-98af-c21b5cbaeeac","Type":"ContainerDied","Data":"84c6f6f7663514af84eeb96927c8cd471205967d5b0e423193fca3306842e012"} Nov 03 15:12:40.109426 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:40.109398 4290 generic.go:356] "Generic (PLEG): container finished" podID="998236af-9ffc-4b16-9a61-5188945ff1d0" containerID="e52325101348e4cc3e3a492594bee08a2b94a5c50b5134558e5bbb050a888478" exitCode=0 Nov 03 15:12:40.109544 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:40.109475 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" event={"ID":"998236af-9ffc-4b16-9a61-5188945ff1d0","Type":"ContainerDied","Data":"e52325101348e4cc3e3a492594bee08a2b94a5c50b5134558e5bbb050a888478"} Nov 03 15:12:40.112617 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:40.112512 4290 generic.go:356] "Generic (PLEG): container finished" podID="03a0fb03-c422-430a-aa4b-6226262bcb73" containerID="127b5048f2877d6ebc23e3f2246dce2df2e6ad8dc344a505271bfdb6f91c7424" exitCode=0 Nov 03 15:12:40.112682 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:40.112625 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" event={"ID":"03a0fb03-c422-430a-aa4b-6226262bcb73","Type":"ContainerDied","Data":"127b5048f2877d6ebc23e3f2246dce2df2e6ad8dc344a505271bfdb6f91c7424"} Nov 03 15:12:40.114850 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:40.114824 4290 generic.go:356] "Generic (PLEG): container finished" podID="9af18a4e-7c82-49f3-9edb-8de433790ea3" containerID="b0c50c3499c34b46850a778dd5877b9f4af326687111a6033f3319ada26b1a0a" exitCode=0 Nov 03 15:12:40.115169 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:40.115093 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" event={"ID":"9af18a4e-7c82-49f3-9edb-8de433790ea3","Type":"ContainerDied","Data":"b0c50c3499c34b46850a778dd5877b9f4af326687111a6033f3319ada26b1a0a"} Nov 03 15:12:40.117839 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:40.117658 4290 generic.go:356] "Generic (PLEG): container finished" podID="778c932f-5f44-476a-b6ca-dd489022e1e6" containerID="c6aec65ed94e0c71e1ff295ec7613f6ff95d70b52c75ebf31397254cb21245cf" exitCode=0 Nov 03 15:12:40.117839 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:40.117753 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" event={"ID":"778c932f-5f44-476a-b6ca-dd489022e1e6","Type":"ContainerDied","Data":"c6aec65ed94e0c71e1ff295ec7613f6ff95d70b52c75ebf31397254cb21245cf"} Nov 03 15:12:41.128373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:41.127613 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" event={"ID":"f351bf29-892f-4c6f-9c1a-b284adb73e58","Type":"ContainerStarted","Data":"5de435b6025b03c602a3bb252f5dedd0e7fe02a5c5ffe34820f1c7555ac4726b"} Nov 03 15:12:41.130590 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:41.130561 4290 generic.go:356] "Generic (PLEG): container finished" podID="bc6a18ed-d1fe-436e-afad-2acfec2954cb" containerID="7ad61ad1bfe9be8e02fe7a49c5d78dccdf0b88f0bc709f6cb4cc9e395102c539" exitCode=0 Nov 03 15:12:41.130741 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:41.130650 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" event={"ID":"bc6a18ed-d1fe-436e-afad-2acfec2954cb","Type":"ContainerDied","Data":"7ad61ad1bfe9be8e02fe7a49c5d78dccdf0b88f0bc709f6cb4cc9e395102c539"} Nov 03 15:12:41.136084 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:41.136059 4290 generic.go:356] "Generic (PLEG): container finished" podID="a529a542-a350-4e78-a685-bb48986be4e1" containerID="7691f8a345c05ba3fb09811184de24b2882d6e9780e7a4325b4bf75b93a47c31" exitCode=0 Nov 03 15:12:41.136213 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:41.136190 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" event={"ID":"a529a542-a350-4e78-a685-bb48986be4e1","Type":"ContainerDied","Data":"7691f8a345c05ba3fb09811184de24b2882d6e9780e7a4325b4bf75b93a47c31"} Nov 03 15:12:41.139067 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:41.139013 4290 generic.go:356] "Generic (PLEG): container finished" podID="5deaed3c-667f-4079-a8ce-a697ada69c7f" containerID="902c620afb802ab59b6ddf5a0cf7245cc85b86b320e24029f57399cc8b1a6017" exitCode=0 Nov 03 15:12:41.139171 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:41.139087 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" event={"ID":"5deaed3c-667f-4079-a8ce-a697ada69c7f","Type":"ContainerDied","Data":"902c620afb802ab59b6ddf5a0cf7245cc85b86b320e24029f57399cc8b1a6017"} Nov 03 15:12:41.140986 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:41.140962 4290 generic.go:356] "Generic (PLEG): container finished" podID="29de4a80-6ad0-4d46-bfbe-4f5b013a879f" containerID="0c360a1f4a4b50c8c52cf3f478b84836c4130555c60feb752373e7295e2cbe50" exitCode=0 Nov 03 15:12:41.141089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:41.141027 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" event={"ID":"29de4a80-6ad0-4d46-bfbe-4f5b013a879f","Type":"ContainerDied","Data":"0c360a1f4a4b50c8c52cf3f478b84836c4130555c60feb752373e7295e2cbe50"} Nov 03 15:12:42.152682 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:42.152588 4290 generic.go:356] "Generic (PLEG): container finished" podID="2c8340b3-582d-4d77-98af-c21b5cbaeeac" containerID="139626a9c91f2f332b10d0e8022456edf00d2c9d79963a8aac9478a4a720f6a0" exitCode=0 Nov 03 15:12:42.153270 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:42.153200 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" event={"ID":"2c8340b3-582d-4d77-98af-c21b5cbaeeac","Type":"ContainerDied","Data":"139626a9c91f2f332b10d0e8022456edf00d2c9d79963a8aac9478a4a720f6a0"} Nov 03 15:12:42.159737 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:42.159677 4290 generic.go:356] "Generic (PLEG): container finished" podID="998236af-9ffc-4b16-9a61-5188945ff1d0" containerID="739d905267b7ebef11c1cb0ba9376d3ef71878c62fab54ce54a359932454d86c" exitCode=0 Nov 03 15:12:42.159834 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:42.159788 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" event={"ID":"998236af-9ffc-4b16-9a61-5188945ff1d0","Type":"ContainerDied","Data":"739d905267b7ebef11c1cb0ba9376d3ef71878c62fab54ce54a359932454d86c"} Nov 03 15:12:42.163573 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:42.162921 4290 generic.go:356] "Generic (PLEG): container finished" podID="03a0fb03-c422-430a-aa4b-6226262bcb73" containerID="f73b5c7ce1aa3f3e31636b3edd35d3dc8f6021478ed705bcd9b1aef425b5adea" exitCode=0 Nov 03 15:12:42.163573 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:42.163226 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" event={"ID":"03a0fb03-c422-430a-aa4b-6226262bcb73","Type":"ContainerDied","Data":"f73b5c7ce1aa3f3e31636b3edd35d3dc8f6021478ed705bcd9b1aef425b5adea"} Nov 03 15:12:42.166039 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:42.166017 4290 generic.go:356] "Generic (PLEG): container finished" podID="f351bf29-892f-4c6f-9c1a-b284adb73e58" containerID="5de435b6025b03c602a3bb252f5dedd0e7fe02a5c5ffe34820f1c7555ac4726b" exitCode=0 Nov 03 15:12:42.166188 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:42.166113 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" event={"ID":"f351bf29-892f-4c6f-9c1a-b284adb73e58","Type":"ContainerDied","Data":"5de435b6025b03c602a3bb252f5dedd0e7fe02a5c5ffe34820f1c7555ac4726b"} Nov 03 15:12:42.169543 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:42.169520 4290 generic.go:356] "Generic (PLEG): container finished" podID="9af18a4e-7c82-49f3-9edb-8de433790ea3" containerID="754ff0553efc56d968fd0e39457391bfde595b12d7ad22c392ee4a54c733979e" exitCode=0 Nov 03 15:12:42.169657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:42.169594 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" event={"ID":"9af18a4e-7c82-49f3-9edb-8de433790ea3","Type":"ContainerDied","Data":"754ff0553efc56d968fd0e39457391bfde595b12d7ad22c392ee4a54c733979e"} Nov 03 15:12:42.174924 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:42.174340 4290 generic.go:356] "Generic (PLEG): container finished" podID="778c932f-5f44-476a-b6ca-dd489022e1e6" containerID="2e342c083a38307268490e149982acbe4ba1970dda257bd9ebcf1f60c361f8fc" exitCode=0 Nov 03 15:12:42.174924 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:42.174463 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" event={"ID":"778c932f-5f44-476a-b6ca-dd489022e1e6","Type":"ContainerDied","Data":"2e342c083a38307268490e149982acbe4ba1970dda257bd9ebcf1f60c361f8fc"} Nov 03 15:12:50.330694 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:50.328231 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" event={"ID":"29de4a80-6ad0-4d46-bfbe-4f5b013a879f","Type":"ContainerStarted","Data":"6d3b8db2db1e63f6311deac4f419117a98c849cfd123031847c90863d41ef356"} Nov 03 15:12:51.353374 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:51.352303 4290 generic.go:356] "Generic (PLEG): container finished" podID="5deaed3c-667f-4079-a8ce-a697ada69c7f" containerID="d91ec7fc915a24d687370c5e234cd5a217224c90df129cf382ed655b8d561e26" exitCode=0 Nov 03 15:12:51.353374 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:51.352400 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" event={"ID":"5deaed3c-667f-4079-a8ce-a697ada69c7f","Type":"ContainerDied","Data":"d91ec7fc915a24d687370c5e234cd5a217224c90df129cf382ed655b8d561e26"} Nov 03 15:12:51.361353 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:51.361086 4290 generic.go:356] "Generic (PLEG): container finished" podID="29de4a80-6ad0-4d46-bfbe-4f5b013a879f" containerID="6d3b8db2db1e63f6311deac4f419117a98c849cfd123031847c90863d41ef356" exitCode=0 Nov 03 15:12:51.361353 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:51.361238 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" event={"ID":"29de4a80-6ad0-4d46-bfbe-4f5b013a879f","Type":"ContainerDied","Data":"6d3b8db2db1e63f6311deac4f419117a98c849cfd123031847c90863d41ef356"} Nov 03 15:12:59.474269 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:59.473251 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" event={"ID":"190d9f1a-5aa8-4281-9a33-65ed716e2754","Type":"ContainerStarted","Data":"513912d1bd008dde3c70119fccc548210479bcd17d071b5639313dc405e1fc61"} Nov 03 15:12:59.597072 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:12:59.597009 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" podStartSLOduration=2.870888861 podStartE2EDuration="45.596990477s" totalImagesPullingTime="42.726101616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:12:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:12:59.596745447 +0000 UTC m=+287.027703254" watchObservedRunningTime="2026-11-03 15:12:59.596990477 +0000 UTC m=+287.027948284" Nov 03 15:13:00.488764 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:00.488142 4290 generic.go:356] "Generic (PLEG): container finished" podID="190d9f1a-5aa8-4281-9a33-65ed716e2754" containerID="513912d1bd008dde3c70119fccc548210479bcd17d071b5639313dc405e1fc61" exitCode=0 Nov 03 15:13:00.488764 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:00.488296 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" event={"ID":"190d9f1a-5aa8-4281-9a33-65ed716e2754","Type":"ContainerDied","Data":"513912d1bd008dde3c70119fccc548210479bcd17d071b5639313dc405e1fc61"} Nov 03 15:13:01.511687 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:01.511602 4290 generic.go:356] "Generic (PLEG): container finished" podID="0d7d6100-6217-4649-a0dc-67879e4b0b79" containerID="a733d51009f00372e83cb16f34dc1720dec46a202ceeee36c11efec996f89477" exitCode=0 Nov 03 15:13:01.512564 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:01.512154 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" event={"ID":"0d7d6100-6217-4649-a0dc-67879e4b0b79","Type":"ContainerDied","Data":"a733d51009f00372e83cb16f34dc1720dec46a202ceeee36c11efec996f89477"} Nov 03 15:13:01.737910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:01.737587 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" Nov 03 15:13:01.851472 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:01.850834 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190d9f1a-5aa8-4281-9a33-65ed716e2754-bundle\" (UniqueName: \"kubernetes.io/empty-dir/190d9f1a-5aa8-4281-9a33-65ed716e2754-bundle\") pod \"190d9f1a-5aa8-4281-9a33-65ed716e2754\" (UID: \"190d9f1a-5aa8-4281-9a33-65ed716e2754\") " Nov 03 15:13:01.851472 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:01.850976 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190d9f1a-5aa8-4281-9a33-65ed716e2754-kube-api-access-k8l72\" (UniqueName: \"kubernetes.io/projected/190d9f1a-5aa8-4281-9a33-65ed716e2754-kube-api-access-k8l72\") pod \"190d9f1a-5aa8-4281-9a33-65ed716e2754\" (UID: \"190d9f1a-5aa8-4281-9a33-65ed716e2754\") " Nov 03 15:13:01.851472 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:01.850996 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190d9f1a-5aa8-4281-9a33-65ed716e2754-util\" (UniqueName: \"kubernetes.io/empty-dir/190d9f1a-5aa8-4281-9a33-65ed716e2754-util\") pod \"190d9f1a-5aa8-4281-9a33-65ed716e2754\" (UID: \"190d9f1a-5aa8-4281-9a33-65ed716e2754\") " Nov 03 15:13:01.852186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:01.852131 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190d9f1a-5aa8-4281-9a33-65ed716e2754-bundle" pod "190d9f1a-5aa8-4281-9a33-65ed716e2754" (UID: "190d9f1a-5aa8-4281-9a33-65ed716e2754"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:01.858141 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:01.858111 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190d9f1a-5aa8-4281-9a33-65ed716e2754-util" pod "190d9f1a-5aa8-4281-9a33-65ed716e2754" (UID: "190d9f1a-5aa8-4281-9a33-65ed716e2754"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:01.868354 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:01.868307 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190d9f1a-5aa8-4281-9a33-65ed716e2754-kube-api-access-k8l72" pod "190d9f1a-5aa8-4281-9a33-65ed716e2754" (UID: "190d9f1a-5aa8-4281-9a33-65ed716e2754"). InnerVolumeSpecName "kube-api-access-k8l72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:13:01.952637 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:01.952532 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8l72\" (UniqueName: \"kubernetes.io/projected/190d9f1a-5aa8-4281-9a33-65ed716e2754-kube-api-access-k8l72\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:01.952637 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:01.952568 4290 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/190d9f1a-5aa8-4281-9a33-65ed716e2754-util\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:01.952637 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:01.952585 4290 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/190d9f1a-5aa8-4281-9a33-65ed716e2754-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:02.535861 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:02.533480 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" Nov 03 15:13:02.535861 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:02.535771 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qpkxd" event={"ID":"190d9f1a-5aa8-4281-9a33-65ed716e2754","Type":"ContainerDied","Data":"48df3f5038d26b57f69587376737503fe6cb9f8517a179889f6e22bfdb27d2e1"} Nov 03 15:13:02.535861 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:02.535830 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48df3f5038d26b57f69587376737503fe6cb9f8517a179889f6e22bfdb27d2e1" Nov 03 15:13:02.824660 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:02.824553 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" Nov 03 15:13:02.964753 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:02.963982 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d7d6100-6217-4649-a0dc-67879e4b0b79-kube-api-access-9z2cp\" (UniqueName: \"kubernetes.io/projected/0d7d6100-6217-4649-a0dc-67879e4b0b79-kube-api-access-9z2cp\") pod \"0d7d6100-6217-4649-a0dc-67879e4b0b79\" (UID: \"0d7d6100-6217-4649-a0dc-67879e4b0b79\") " Nov 03 15:13:02.964753 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:02.964044 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7d6100-6217-4649-a0dc-67879e4b0b79-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d7d6100-6217-4649-a0dc-67879e4b0b79-bundle\") pod \"0d7d6100-6217-4649-a0dc-67879e4b0b79\" (UID: \"0d7d6100-6217-4649-a0dc-67879e4b0b79\") " Nov 03 15:13:02.964753 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:02.964090 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7d6100-6217-4649-a0dc-67879e4b0b79-util\" (UniqueName: \"kubernetes.io/empty-dir/0d7d6100-6217-4649-a0dc-67879e4b0b79-util\") pod \"0d7d6100-6217-4649-a0dc-67879e4b0b79\" (UID: \"0d7d6100-6217-4649-a0dc-67879e4b0b79\") " Nov 03 15:13:02.968930 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:02.968888 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7d6100-6217-4649-a0dc-67879e4b0b79-bundle" pod "0d7d6100-6217-4649-a0dc-67879e4b0b79" (UID: "0d7d6100-6217-4649-a0dc-67879e4b0b79"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:02.973056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:02.973031 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7d6100-6217-4649-a0dc-67879e4b0b79-util" pod "0d7d6100-6217-4649-a0dc-67879e4b0b79" (UID: "0d7d6100-6217-4649-a0dc-67879e4b0b79"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:02.979942 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:02.979911 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7d6100-6217-4649-a0dc-67879e4b0b79-kube-api-access-9z2cp" pod "0d7d6100-6217-4649-a0dc-67879e4b0b79" (UID: "0d7d6100-6217-4649-a0dc-67879e4b0b79"). InnerVolumeSpecName "kube-api-access-9z2cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:13:03.065843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:03.065439 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z2cp\" (UniqueName: \"kubernetes.io/projected/0d7d6100-6217-4649-a0dc-67879e4b0b79-kube-api-access-9z2cp\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:03.065843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:03.065473 4290 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d7d6100-6217-4649-a0dc-67879e4b0b79-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:03.065843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:03.065488 4290 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d7d6100-6217-4649-a0dc-67879e4b0b79-util\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:03.555587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:03.555353 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" event={"ID":"0d7d6100-6217-4649-a0dc-67879e4b0b79","Type":"ContainerDied","Data":"086a0ca300b0f4a5682fe3e81f731e8faff7c05b43887bdc543502ddb3f2b494"} Nov 03 15:13:03.555587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:03.555394 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="086a0ca300b0f4a5682fe3e81f731e8faff7c05b43887bdc543502ddb3f2b494" Nov 03 15:13:03.555587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:03.555471 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pvgdc" Nov 03 15:13:05.592280 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:05.592231 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" event={"ID":"bc6a18ed-d1fe-436e-afad-2acfec2954cb","Type":"ContainerStarted","Data":"f400fa42555e8d2750fb59186151e4b7de6d8caebab37a2501c9e1e041e56e76"} Nov 03 15:13:05.596241 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:05.596208 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" event={"ID":"a529a542-a350-4e78-a685-bb48986be4e1","Type":"ContainerStarted","Data":"5e459d664776879f3dcc6eb9cfd4ed4babdf57120fc7a06747916d4423b508b1"} Nov 03 15:13:05.599430 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:05.599310 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" event={"ID":"29de4a80-6ad0-4d46-bfbe-4f5b013a879f","Type":"ContainerStarted","Data":"808c7b91a06afd4096ca660586a1164c8d7c7b464d949ea94f8db786b5e95987"} Nov 03 15:13:05.839940 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:05.839169 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" podStartSLOduration=3.381014716 podStartE2EDuration="43.839152103s" totalImagesPullingTime="40.458137387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:12:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:05.719605762 +0000 UTC m=+293.150563570" watchObservedRunningTime="2026-11-03 15:13:05.839152103 +0000 UTC m=+293.270109906" Nov 03 15:13:05.842092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:05.841016 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" podStartSLOduration=3.210291146 podStartE2EDuration="43.841007325s" totalImagesPullingTime="40.630716179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:12:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:05.839085373 +0000 UTC m=+293.270043180" watchObservedRunningTime="2026-11-03 15:13:05.841007325 +0000 UTC m=+293.271965131" Nov 03 15:13:06.606506 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.606475 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" event={"ID":"778c932f-5f44-476a-b6ca-dd489022e1e6","Type":"ContainerStarted","Data":"4113bb41c5fb0289fa8347831bea703b0bc2c86f40071ea04522946ef698014a"} Nov 03 15:13:06.608870 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.608843 4290 generic.go:356] "Generic (PLEG): container finished" podID="bc6a18ed-d1fe-436e-afad-2acfec2954cb" containerID="f400fa42555e8d2750fb59186151e4b7de6d8caebab37a2501c9e1e041e56e76" exitCode=0 Nov 03 15:13:06.609005 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.608890 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" event={"ID":"bc6a18ed-d1fe-436e-afad-2acfec2954cb","Type":"ContainerDied","Data":"f400fa42555e8d2750fb59186151e4b7de6d8caebab37a2501c9e1e041e56e76"} Nov 03 15:13:06.611002 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.610978 4290 generic.go:356] "Generic (PLEG): container finished" podID="a529a542-a350-4e78-a685-bb48986be4e1" containerID="5e459d664776879f3dcc6eb9cfd4ed4babdf57120fc7a06747916d4423b508b1" exitCode=0 Nov 03 15:13:06.611122 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.611060 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" event={"ID":"a529a542-a350-4e78-a685-bb48986be4e1","Type":"ContainerDied","Data":"5e459d664776879f3dcc6eb9cfd4ed4babdf57120fc7a06747916d4423b508b1"} Nov 03 15:13:06.613039 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.613020 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" event={"ID":"5deaed3c-667f-4079-a8ce-a697ada69c7f","Type":"ContainerStarted","Data":"185aec78698005f3106e057c144e563501fe57430a33fb5d4242f052d91bb9f0"} Nov 03 15:13:06.614944 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.614913 4290 generic.go:356] "Generic (PLEG): container finished" podID="29de4a80-6ad0-4d46-bfbe-4f5b013a879f" containerID="808c7b91a06afd4096ca660586a1164c8d7c7b464d949ea94f8db786b5e95987" exitCode=0 Nov 03 15:13:06.615024 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.614957 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" event={"ID":"29de4a80-6ad0-4d46-bfbe-4f5b013a879f","Type":"ContainerDied","Data":"808c7b91a06afd4096ca660586a1164c8d7c7b464d949ea94f8db786b5e95987"} Nov 03 15:13:06.730321 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.730277 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" podStartSLOduration=3.001339684 podStartE2EDuration="50.730262226s" totalImagesPullingTime="47.728922542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:12:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:06.727787577 +0000 UTC m=+294.158745384" watchObservedRunningTime="2026-11-03 15:13:06.730262226 +0000 UTC m=+294.161220032" Nov 03 15:13:06.742960 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.742909 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" Nov 03 15:13:06.906787 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.906755 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-kube-api-access-jkcjk\" (UniqueName: \"kubernetes.io/projected/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-kube-api-access-jkcjk\") pod \"29de4a80-6ad0-4d46-bfbe-4f5b013a879f\" (UID: \"29de4a80-6ad0-4d46-bfbe-4f5b013a879f\") " Nov 03 15:13:06.906960 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.906857 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-util\" (UniqueName: \"kubernetes.io/empty-dir/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-util\") pod \"29de4a80-6ad0-4d46-bfbe-4f5b013a879f\" (UID: \"29de4a80-6ad0-4d46-bfbe-4f5b013a879f\") " Nov 03 15:13:06.906960 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.906903 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-bundle\") pod \"29de4a80-6ad0-4d46-bfbe-4f5b013a879f\" (UID: \"29de4a80-6ad0-4d46-bfbe-4f5b013a879f\") " Nov 03 15:13:06.907277 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.907253 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-bundle" pod "29de4a80-6ad0-4d46-bfbe-4f5b013a879f" (UID: "29de4a80-6ad0-4d46-bfbe-4f5b013a879f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:06.909022 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.908997 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-kube-api-access-jkcjk" pod "29de4a80-6ad0-4d46-bfbe-4f5b013a879f" (UID: "29de4a80-6ad0-4d46-bfbe-4f5b013a879f"). InnerVolumeSpecName "kube-api-access-jkcjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:13:06.911790 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.911767 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-util" pod "29de4a80-6ad0-4d46-bfbe-4f5b013a879f" (UID: "29de4a80-6ad0-4d46-bfbe-4f5b013a879f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:06.965610 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:06.965519 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" podStartSLOduration=4.739844365 podStartE2EDuration="51.965507731s" totalImagesPullingTime="47.225663366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:12:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:06.964554929 +0000 UTC m=+294.395512749" watchObservedRunningTime="2026-11-03 15:13:06.965507731 +0000 UTC m=+294.396465536" Nov 03 15:13:07.008402 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.008373 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkcjk\" (UniqueName: \"kubernetes.io/projected/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-kube-api-access-jkcjk\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:07.008402 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.008398 4290 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-util\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:07.008402 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.008407 4290 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29de4a80-6ad0-4d46-bfbe-4f5b013a879f-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:07.622501 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.622472 4290 generic.go:356] "Generic (PLEG): container finished" podID="5deaed3c-667f-4079-a8ce-a697ada69c7f" containerID="185aec78698005f3106e057c144e563501fe57430a33fb5d4242f052d91bb9f0" exitCode=0 Nov 03 15:13:07.622965 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.622562 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" event={"ID":"5deaed3c-667f-4079-a8ce-a697ada69c7f","Type":"ContainerDied","Data":"185aec78698005f3106e057c144e563501fe57430a33fb5d4242f052d91bb9f0"} Nov 03 15:13:07.624855 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.624823 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" event={"ID":"29de4a80-6ad0-4d46-bfbe-4f5b013a879f","Type":"ContainerDied","Data":"1af4fe8da6a7085d95de6fe25fd33d0abfffba33fcfda17d1ef3a813d0f0c724"} Nov 03 15:13:07.624855 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.624848 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kprpv" Nov 03 15:13:07.625031 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.624880 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1af4fe8da6a7085d95de6fe25fd33d0abfffba33fcfda17d1ef3a813d0f0c724" Nov 03 15:13:07.629686 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.629663 4290 generic.go:356] "Generic (PLEG): container finished" podID="2c8340b3-582d-4d77-98af-c21b5cbaeeac" containerID="b89086bbb0fe9447f12b84b8ff6c7cf28b1aff26b4b60bc1adad1fec41d3b71a" exitCode=0 Nov 03 15:13:07.629820 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.629740 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" event={"ID":"2c8340b3-582d-4d77-98af-c21b5cbaeeac","Type":"ContainerDied","Data":"b89086bbb0fe9447f12b84b8ff6c7cf28b1aff26b4b60bc1adad1fec41d3b71a"} Nov 03 15:13:07.631800 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.631773 4290 generic.go:356] "Generic (PLEG): container finished" podID="998236af-9ffc-4b16-9a61-5188945ff1d0" containerID="86bf8fde73056a3657b78940ff35f8cc1f13bba5464c9e5ed6c87e35719cf863" exitCode=0 Nov 03 15:13:07.631914 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.631834 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" event={"ID":"998236af-9ffc-4b16-9a61-5188945ff1d0","Type":"ContainerDied","Data":"86bf8fde73056a3657b78940ff35f8cc1f13bba5464c9e5ed6c87e35719cf863"} Nov 03 15:13:07.633946 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.633929 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:13:07.634647 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.634625 4290 generic.go:356] "Generic (PLEG): container finished" podID="03a0fb03-c422-430a-aa4b-6226262bcb73" containerID="81040a2176104fe0c98fa3b4310ec52115a64659d35d3a134ec405ab576105b0" exitCode=1 Nov 03 15:13:07.634771 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.634720 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" event={"ID":"03a0fb03-c422-430a-aa4b-6226262bcb73","Type":"ContainerDied","Data":"81040a2176104fe0c98fa3b4310ec52115a64659d35d3a134ec405ab576105b0"} Nov 03 15:13:07.636499 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.636478 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:13:07.637130 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.637109 4290 generic.go:356] "Generic (PLEG): container finished" podID="f351bf29-892f-4c6f-9c1a-b284adb73e58" containerID="d44e2e8919c4b0b00cdf94f08c4cbee95a381b24f75e93eb168772d02c68ef6e" exitCode=1 Nov 03 15:13:07.637241 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.637184 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" event={"ID":"f351bf29-892f-4c6f-9c1a-b284adb73e58","Type":"ContainerDied","Data":"d44e2e8919c4b0b00cdf94f08c4cbee95a381b24f75e93eb168772d02c68ef6e"} Nov 03 15:13:07.639122 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.639099 4290 generic.go:356] "Generic (PLEG): container finished" podID="9af18a4e-7c82-49f3-9edb-8de433790ea3" containerID="e79d04a8154d47c41963b55d8b81ed87b3f97654819e06fea189fa6b50c2ae25" exitCode=0 Nov 03 15:13:07.639224 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.639126 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" event={"ID":"9af18a4e-7c82-49f3-9edb-8de433790ea3","Type":"ContainerDied","Data":"e79d04a8154d47c41963b55d8b81ed87b3f97654819e06fea189fa6b50c2ae25"} Nov 03 15:13:07.641038 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.641020 4290 generic.go:356] "Generic (PLEG): container finished" podID="778c932f-5f44-476a-b6ca-dd489022e1e6" containerID="4113bb41c5fb0289fa8347831bea703b0bc2c86f40071ea04522946ef698014a" exitCode=0 Nov 03 15:13:07.641143 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.641044 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" event={"ID":"778c932f-5f44-476a-b6ca-dd489022e1e6","Type":"ContainerDied","Data":"4113bb41c5fb0289fa8347831bea703b0bc2c86f40071ea04522946ef698014a"} Nov 03 15:13:07.766435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.765686 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" Nov 03 15:13:07.815014 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.814978 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" Nov 03 15:13:07.916822 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.916740 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc6a18ed-d1fe-436e-afad-2acfec2954cb-util\" (UniqueName: \"kubernetes.io/empty-dir/bc6a18ed-d1fe-436e-afad-2acfec2954cb-util\") pod \"bc6a18ed-d1fe-436e-afad-2acfec2954cb\" (UID: \"bc6a18ed-d1fe-436e-afad-2acfec2954cb\") " Nov 03 15:13:07.916822 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.916815 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a529a542-a350-4e78-a685-bb48986be4e1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a529a542-a350-4e78-a685-bb48986be4e1-bundle\") pod \"a529a542-a350-4e78-a685-bb48986be4e1\" (UID: \"a529a542-a350-4e78-a685-bb48986be4e1\") " Nov 03 15:13:07.917056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.916885 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a529a542-a350-4e78-a685-bb48986be4e1-util\" (UniqueName: \"kubernetes.io/empty-dir/a529a542-a350-4e78-a685-bb48986be4e1-util\") pod \"a529a542-a350-4e78-a685-bb48986be4e1\" (UID: \"a529a542-a350-4e78-a685-bb48986be4e1\") " Nov 03 15:13:07.917056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.916934 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a529a542-a350-4e78-a685-bb48986be4e1-kube-api-access-vrx7d\" (UniqueName: \"kubernetes.io/projected/a529a542-a350-4e78-a685-bb48986be4e1-kube-api-access-vrx7d\") pod \"a529a542-a350-4e78-a685-bb48986be4e1\" (UID: \"a529a542-a350-4e78-a685-bb48986be4e1\") " Nov 03 15:13:07.917056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.916971 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc6a18ed-d1fe-436e-afad-2acfec2954cb-kube-api-access-gmbgw\" (UniqueName: \"kubernetes.io/projected/bc6a18ed-d1fe-436e-afad-2acfec2954cb-kube-api-access-gmbgw\") pod \"bc6a18ed-d1fe-436e-afad-2acfec2954cb\" (UID: \"bc6a18ed-d1fe-436e-afad-2acfec2954cb\") " Nov 03 15:13:07.917056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.917009 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc6a18ed-d1fe-436e-afad-2acfec2954cb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc6a18ed-d1fe-436e-afad-2acfec2954cb-bundle\") pod \"bc6a18ed-d1fe-436e-afad-2acfec2954cb\" (UID: \"bc6a18ed-d1fe-436e-afad-2acfec2954cb\") " Nov 03 15:13:07.917644 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.917617 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a529a542-a350-4e78-a685-bb48986be4e1-bundle" pod "a529a542-a350-4e78-a685-bb48986be4e1" (UID: "a529a542-a350-4e78-a685-bb48986be4e1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:07.917755 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.917670 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6a18ed-d1fe-436e-afad-2acfec2954cb-bundle" pod "bc6a18ed-d1fe-436e-afad-2acfec2954cb" (UID: "bc6a18ed-d1fe-436e-afad-2acfec2954cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:07.919622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.919598 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a529a542-a350-4e78-a685-bb48986be4e1-kube-api-access-vrx7d" pod "a529a542-a350-4e78-a685-bb48986be4e1" (UID: "a529a542-a350-4e78-a685-bb48986be4e1"). InnerVolumeSpecName "kube-api-access-vrx7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:13:07.919792 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.919678 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6a18ed-d1fe-436e-afad-2acfec2954cb-kube-api-access-gmbgw" pod "bc6a18ed-d1fe-436e-afad-2acfec2954cb" (UID: "bc6a18ed-d1fe-436e-afad-2acfec2954cb"). InnerVolumeSpecName "kube-api-access-gmbgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:13:07.923296 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.923271 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6a18ed-d1fe-436e-afad-2acfec2954cb-util" pod "bc6a18ed-d1fe-436e-afad-2acfec2954cb" (UID: "bc6a18ed-d1fe-436e-afad-2acfec2954cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:07.923370 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:07.923292 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a529a542-a350-4e78-a685-bb48986be4e1-util" pod "a529a542-a350-4e78-a685-bb48986be4e1" (UID: "a529a542-a350-4e78-a685-bb48986be4e1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:08.018537 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.018509 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmbgw\" (UniqueName: \"kubernetes.io/projected/bc6a18ed-d1fe-436e-afad-2acfec2954cb-kube-api-access-gmbgw\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:08.018537 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.018534 4290 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc6a18ed-d1fe-436e-afad-2acfec2954cb-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:08.018671 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.018543 4290 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc6a18ed-d1fe-436e-afad-2acfec2954cb-util\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:08.018671 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.018552 4290 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a529a542-a350-4e78-a685-bb48986be4e1-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:08.018671 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.018559 4290 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a529a542-a350-4e78-a685-bb48986be4e1-util\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:08.018671 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.018567 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrx7d\" (UniqueName: \"kubernetes.io/projected/a529a542-a350-4e78-a685-bb48986be4e1-kube-api-access-vrx7d\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:08.653073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.653031 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" event={"ID":"bc6a18ed-d1fe-436e-afad-2acfec2954cb","Type":"ContainerDied","Data":"fc7cba88187722f83380dadda034af391db367c92524954b04efee1879d37077"} Nov 03 15:13:08.653073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.653072 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgwd4z" Nov 03 15:13:08.653535 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.653120 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc7cba88187722f83380dadda034af391db367c92524954b04efee1879d37077" Nov 03 15:13:08.656531 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.656505 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" Nov 03 15:13:08.657527 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.656795 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4xs2" event={"ID":"a529a542-a350-4e78-a685-bb48986be4e1","Type":"ContainerDied","Data":"3f975713e5d9e6e59e36ff1e66889928a2122bf6de0e99dd3bee364fb5fddc2a"} Nov 03 15:13:08.658648 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.658610 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f975713e5d9e6e59e36ff1e66889928a2122bf6de0e99dd3bee364fb5fddc2a" Nov 03 15:13:08.738324 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.738239 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp"] Nov 03 15:13:08.738975 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.738942 4290 cpu_manager.go:414] "containerMap: removing container" podUID="a529a542-a350-4e78-a685-bb48986be4e1" containerName="util" Nov 03 15:13:08.739088 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.739077 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a529a542-a350-4e78-a685-bb48986be4e1" containerName="util" Nov 03 15:13:08.739165 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.739157 4290 cpu_manager.go:414] "containerMap: removing container" podUID="29de4a80-6ad0-4d46-bfbe-4f5b013a879f" containerName="util" Nov 03 15:13:08.739229 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.739209 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29de4a80-6ad0-4d46-bfbe-4f5b013a879f" containerName="util" Nov 03 15:13:08.739299 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.739288 4290 cpu_manager.go:414] "containerMap: removing container" podUID="29de4a80-6ad0-4d46-bfbe-4f5b013a879f" containerName="extract" Nov 03 15:13:08.739349 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.739342 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29de4a80-6ad0-4d46-bfbe-4f5b013a879f" containerName="extract" Nov 03 15:13:08.739417 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.739410 4290 cpu_manager.go:414] "containerMap: removing container" podUID="0d7d6100-6217-4649-a0dc-67879e4b0b79" containerName="pull" Nov 03 15:13:08.739486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.739477 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7d6100-6217-4649-a0dc-67879e4b0b79" containerName="pull" Nov 03 15:13:08.739545 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.739539 4290 cpu_manager.go:414] "containerMap: removing container" podUID="190d9f1a-5aa8-4281-9a33-65ed716e2754" containerName="pull" Nov 03 15:13:08.739591 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.739584 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190d9f1a-5aa8-4281-9a33-65ed716e2754" containerName="pull" Nov 03 15:13:08.739661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.739654 4290 cpu_manager.go:414] "containerMap: removing container" podUID="190d9f1a-5aa8-4281-9a33-65ed716e2754" containerName="util" Nov 03 15:13:08.739738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.739728 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190d9f1a-5aa8-4281-9a33-65ed716e2754" containerName="util" Nov 03 15:13:08.739832 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.739822 4290 cpu_manager.go:414] "containerMap: removing container" podUID="a529a542-a350-4e78-a685-bb48986be4e1" containerName="pull" Nov 03 15:13:08.740789 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.740764 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a529a542-a350-4e78-a685-bb48986be4e1" containerName="pull" Nov 03 15:13:08.740949 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.740937 4290 cpu_manager.go:414] "containerMap: removing container" podUID="29de4a80-6ad0-4d46-bfbe-4f5b013a879f" containerName="pull" Nov 03 15:13:08.741034 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.741023 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29de4a80-6ad0-4d46-bfbe-4f5b013a879f" containerName="pull" Nov 03 15:13:08.741126 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.741114 4290 cpu_manager.go:414] "containerMap: removing container" podUID="0d7d6100-6217-4649-a0dc-67879e4b0b79" containerName="util" Nov 03 15:13:08.741205 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.741195 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7d6100-6217-4649-a0dc-67879e4b0b79" containerName="util" Nov 03 15:13:08.741283 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.741274 4290 cpu_manager.go:414] "containerMap: removing container" podUID="bc6a18ed-d1fe-436e-afad-2acfec2954cb" containerName="util" Nov 03 15:13:08.741363 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.741353 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc6a18ed-d1fe-436e-afad-2acfec2954cb" containerName="util" Nov 03 15:13:08.741437 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.741429 4290 cpu_manager.go:414] "containerMap: removing container" podUID="190d9f1a-5aa8-4281-9a33-65ed716e2754" containerName="extract" Nov 03 15:13:08.741511 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.741500 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190d9f1a-5aa8-4281-9a33-65ed716e2754" containerName="extract" Nov 03 15:13:08.741578 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.741569 4290 cpu_manager.go:414] "containerMap: removing container" podUID="0d7d6100-6217-4649-a0dc-67879e4b0b79" containerName="extract" Nov 03 15:13:08.741658 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.741647 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7d6100-6217-4649-a0dc-67879e4b0b79" containerName="extract" Nov 03 15:13:08.741758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.741748 4290 cpu_manager.go:414] "containerMap: removing container" podUID="a529a542-a350-4e78-a685-bb48986be4e1" containerName="extract" Nov 03 15:13:08.741843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.741832 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a529a542-a350-4e78-a685-bb48986be4e1" containerName="extract" Nov 03 15:13:08.741930 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.741920 4290 cpu_manager.go:414] "containerMap: removing container" podUID="bc6a18ed-d1fe-436e-afad-2acfec2954cb" containerName="pull" Nov 03 15:13:08.741997 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.741987 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc6a18ed-d1fe-436e-afad-2acfec2954cb" containerName="pull" Nov 03 15:13:08.742100 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.742088 4290 cpu_manager.go:414] "containerMap: removing container" podUID="bc6a18ed-d1fe-436e-afad-2acfec2954cb" containerName="extract" Nov 03 15:13:08.742354 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.742325 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc6a18ed-d1fe-436e-afad-2acfec2954cb" containerName="extract" Nov 03 15:13:08.742617 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.742594 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc6a18ed-d1fe-436e-afad-2acfec2954cb" containerName="extract" Nov 03 15:13:08.742768 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.742755 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="29de4a80-6ad0-4d46-bfbe-4f5b013a879f" containerName="extract" Nov 03 15:13:08.742897 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.742884 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="a529a542-a350-4e78-a685-bb48986be4e1" containerName="extract" Nov 03 15:13:08.743005 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.742994 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7d6100-6217-4649-a0dc-67879e4b0b79" containerName="extract" Nov 03 15:13:08.743103 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.743085 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="190d9f1a-5aa8-4281-9a33-65ed716e2754" containerName="extract" Nov 03 15:13:08.744082 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:13:08.744039 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:13:08.749454 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.749435 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.808285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.807311 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-nkgmk\"" Nov 03 15:13:08.808285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.807813 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 15:13:08.808285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.807872 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 15:13:08.808285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.807980 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 15:13:08.808285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.808034 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 15:13:08.808285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.808053 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 15:13:08.830670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.830061 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecd436a-2f5e-4119-b25b-66902c0140b8-tmp\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.830670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.830156 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ecd436a-2f5e-4119-b25b-66902c0140b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.830670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.830211 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ecd436a-2f5e-4119-b25b-66902c0140b8-manifests\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.830670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.830251 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ecd436a-2f5e-4119-b25b-66902c0140b8-metrics-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.830670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.830284 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ecd436a-2f5e-4119-b25b-66902c0140b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.830670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.830318 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbl25\" (UniqueName: \"kubernetes.io/projected/9ecd436a-2f5e-4119-b25b-66902c0140b8-kube-api-access-xbl25\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.854482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.854454 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp"] Nov 03 15:13:08.859664 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.859643 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" Nov 03 15:13:08.933224 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.933198 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" Nov 03 15:13:08.933769 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.933747 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ecd436a-2f5e-4119-b25b-66902c0140b8-manifests\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.933890 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.933790 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ecd436a-2f5e-4119-b25b-66902c0140b8-metrics-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.933890 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.933823 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ecd436a-2f5e-4119-b25b-66902c0140b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.933890 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.933849 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbl25\" (UniqueName: \"kubernetes.io/projected/9ecd436a-2f5e-4119-b25b-66902c0140b8-kube-api-access-xbl25\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.934049 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.933925 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecd436a-2f5e-4119-b25b-66902c0140b8-tmp\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.934049 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.934007 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ecd436a-2f5e-4119-b25b-66902c0140b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.937500 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.937469 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ecd436a-2f5e-4119-b25b-66902c0140b8-manifests\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.938603 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.938579 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ecd436a-2f5e-4119-b25b-66902c0140b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.939991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.939972 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:13:08.941477 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.941459 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" Nov 03 15:13:08.942391 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.942365 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecd436a-2f5e-4119-b25b-66902c0140b8-tmp\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.942915 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.942896 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ecd436a-2f5e-4119-b25b-66902c0140b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.944435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.944415 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ecd436a-2f5e-4119-b25b-66902c0140b8-metrics-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:08.967614 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.967594 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" Nov 03 15:13:08.970220 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.970200 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:13:08.971849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.971830 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" Nov 03 15:13:08.975061 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.975045 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" Nov 03 15:13:08.978236 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.978221 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" Nov 03 15:13:08.998433 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:08.998374 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbl25\" (UniqueName: \"kubernetes.io/projected/9ecd436a-2f5e-4119-b25b-66902c0140b8-kube-api-access-xbl25\") pod \"opendatahub-operator-controller-manager-84b7cd6487-tgdcp\" (UID: \"9ecd436a-2f5e-4119-b25b-66902c0140b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:09.034556 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.034528 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778c932f-5f44-476a-b6ca-dd489022e1e6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/778c932f-5f44-476a-b6ca-dd489022e1e6-bundle\") pod \"778c932f-5f44-476a-b6ca-dd489022e1e6\" (UID: \"778c932f-5f44-476a-b6ca-dd489022e1e6\") " Nov 03 15:13:09.034658 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.034570 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778c932f-5f44-476a-b6ca-dd489022e1e6-util\" (UniqueName: \"kubernetes.io/empty-dir/778c932f-5f44-476a-b6ca-dd489022e1e6-util\") pod \"778c932f-5f44-476a-b6ca-dd489022e1e6\" (UID: \"778c932f-5f44-476a-b6ca-dd489022e1e6\") " Nov 03 15:13:09.034658 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.034600 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998236af-9ffc-4b16-9a61-5188945ff1d0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/998236af-9ffc-4b16-9a61-5188945ff1d0-bundle\") pod \"998236af-9ffc-4b16-9a61-5188945ff1d0\" (UID: \"998236af-9ffc-4b16-9a61-5188945ff1d0\") " Nov 03 15:13:09.034658 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.034621 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778c932f-5f44-476a-b6ca-dd489022e1e6-kube-api-access-kml4m\" (UniqueName: \"kubernetes.io/projected/778c932f-5f44-476a-b6ca-dd489022e1e6-kube-api-access-kml4m\") pod \"778c932f-5f44-476a-b6ca-dd489022e1e6\" (UID: \"778c932f-5f44-476a-b6ca-dd489022e1e6\") " Nov 03 15:13:09.034658 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.034639 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998236af-9ffc-4b16-9a61-5188945ff1d0-kube-api-access-g9zqf\" (UniqueName: \"kubernetes.io/projected/998236af-9ffc-4b16-9a61-5188945ff1d0-kube-api-access-g9zqf\") pod \"998236af-9ffc-4b16-9a61-5188945ff1d0\" (UID: \"998236af-9ffc-4b16-9a61-5188945ff1d0\") " Nov 03 15:13:09.034901 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.034672 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f351bf29-892f-4c6f-9c1a-b284adb73e58-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f351bf29-892f-4c6f-9c1a-b284adb73e58-bundle\") pod \"f351bf29-892f-4c6f-9c1a-b284adb73e58\" (UID: \"f351bf29-892f-4c6f-9c1a-b284adb73e58\") " Nov 03 15:13:09.034901 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.034725 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f351bf29-892f-4c6f-9c1a-b284adb73e58-util\" (UniqueName: \"kubernetes.io/empty-dir/f351bf29-892f-4c6f-9c1a-b284adb73e58-util\") pod \"f351bf29-892f-4c6f-9c1a-b284adb73e58\" (UID: \"f351bf29-892f-4c6f-9c1a-b284adb73e58\") " Nov 03 15:13:09.034901 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.034795 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f351bf29-892f-4c6f-9c1a-b284adb73e58-kube-api-access-vt682\" (UniqueName: \"kubernetes.io/projected/f351bf29-892f-4c6f-9c1a-b284adb73e58-kube-api-access-vt682\") pod \"f351bf29-892f-4c6f-9c1a-b284adb73e58\" (UID: \"f351bf29-892f-4c6f-9c1a-b284adb73e58\") " Nov 03 15:13:09.034901 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.034855 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998236af-9ffc-4b16-9a61-5188945ff1d0-util\" (UniqueName: \"kubernetes.io/empty-dir/998236af-9ffc-4b16-9a61-5188945ff1d0-util\") pod \"998236af-9ffc-4b16-9a61-5188945ff1d0\" (UID: \"998236af-9ffc-4b16-9a61-5188945ff1d0\") " Nov 03 15:13:09.035738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.035290 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f351bf29-892f-4c6f-9c1a-b284adb73e58-bundle" pod "f351bf29-892f-4c6f-9c1a-b284adb73e58" (UID: "f351bf29-892f-4c6f-9c1a-b284adb73e58"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:09.035738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.035657 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998236af-9ffc-4b16-9a61-5188945ff1d0-bundle" pod "998236af-9ffc-4b16-9a61-5188945ff1d0" (UID: "998236af-9ffc-4b16-9a61-5188945ff1d0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:09.036012 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.035986 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778c932f-5f44-476a-b6ca-dd489022e1e6-bundle" pod "778c932f-5f44-476a-b6ca-dd489022e1e6" (UID: "778c932f-5f44-476a-b6ca-dd489022e1e6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:09.037278 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.037243 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998236af-9ffc-4b16-9a61-5188945ff1d0-kube-api-access-g9zqf" pod "998236af-9ffc-4b16-9a61-5188945ff1d0" (UID: "998236af-9ffc-4b16-9a61-5188945ff1d0"). InnerVolumeSpecName "kube-api-access-g9zqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:13:09.037361 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.037347 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778c932f-5f44-476a-b6ca-dd489022e1e6-kube-api-access-kml4m" pod "778c932f-5f44-476a-b6ca-dd489022e1e6" (UID: "778c932f-5f44-476a-b6ca-dd489022e1e6"). InnerVolumeSpecName "kube-api-access-kml4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:13:09.038389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.038369 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f351bf29-892f-4c6f-9c1a-b284adb73e58-kube-api-access-vt682" pod "f351bf29-892f-4c6f-9c1a-b284adb73e58" (UID: "f351bf29-892f-4c6f-9c1a-b284adb73e58"). InnerVolumeSpecName "kube-api-access-vt682". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:13:09.041997 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.041973 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f351bf29-892f-4c6f-9c1a-b284adb73e58-util" pod "f351bf29-892f-4c6f-9c1a-b284adb73e58" (UID: "f351bf29-892f-4c6f-9c1a-b284adb73e58"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:09.042150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.042127 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778c932f-5f44-476a-b6ca-dd489022e1e6-util" pod "778c932f-5f44-476a-b6ca-dd489022e1e6" (UID: "778c932f-5f44-476a-b6ca-dd489022e1e6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:09.042209 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.042177 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998236af-9ffc-4b16-9a61-5188945ff1d0-util" pod "998236af-9ffc-4b16-9a61-5188945ff1d0" (UID: "998236af-9ffc-4b16-9a61-5188945ff1d0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:09.084281 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.084256 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:09.136222 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.136189 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af18a4e-7c82-49f3-9edb-8de433790ea3-kube-api-access-sxrtd\" (UniqueName: \"kubernetes.io/projected/9af18a4e-7c82-49f3-9edb-8de433790ea3-kube-api-access-sxrtd\") pod \"9af18a4e-7c82-49f3-9edb-8de433790ea3\" (UID: \"9af18a4e-7c82-49f3-9edb-8de433790ea3\") " Nov 03 15:13:09.136345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.136241 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af18a4e-7c82-49f3-9edb-8de433790ea3-util\" (UniqueName: \"kubernetes.io/empty-dir/9af18a4e-7c82-49f3-9edb-8de433790ea3-util\") pod \"9af18a4e-7c82-49f3-9edb-8de433790ea3\" (UID: \"9af18a4e-7c82-49f3-9edb-8de433790ea3\") " Nov 03 15:13:09.136345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.136276 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8340b3-582d-4d77-98af-c21b5cbaeeac-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c8340b3-582d-4d77-98af-c21b5cbaeeac-bundle\") pod \"2c8340b3-582d-4d77-98af-c21b5cbaeeac\" (UID: \"2c8340b3-582d-4d77-98af-c21b5cbaeeac\") " Nov 03 15:13:09.136345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.136315 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a0fb03-c422-430a-aa4b-6226262bcb73-kube-api-access-46p8b\" (UniqueName: \"kubernetes.io/projected/03a0fb03-c422-430a-aa4b-6226262bcb73-kube-api-access-46p8b\") pod \"03a0fb03-c422-430a-aa4b-6226262bcb73\" (UID: \"03a0fb03-c422-430a-aa4b-6226262bcb73\") " Nov 03 15:13:09.136472 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.136358 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8340b3-582d-4d77-98af-c21b5cbaeeac-util\" (UniqueName: \"kubernetes.io/empty-dir/2c8340b3-582d-4d77-98af-c21b5cbaeeac-util\") pod \"2c8340b3-582d-4d77-98af-c21b5cbaeeac\" (UID: \"2c8340b3-582d-4d77-98af-c21b5cbaeeac\") " Nov 03 15:13:09.136472 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.136405 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5deaed3c-667f-4079-a8ce-a697ada69c7f-kube-api-access-nx275\" (UniqueName: \"kubernetes.io/projected/5deaed3c-667f-4079-a8ce-a697ada69c7f-kube-api-access-nx275\") pod \"5deaed3c-667f-4079-a8ce-a697ada69c7f\" (UID: \"5deaed3c-667f-4079-a8ce-a697ada69c7f\") " Nov 03 15:13:09.136534 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.136478 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5deaed3c-667f-4079-a8ce-a697ada69c7f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5deaed3c-667f-4079-a8ce-a697ada69c7f-bundle\") pod \"5deaed3c-667f-4079-a8ce-a697ada69c7f\" (UID: \"5deaed3c-667f-4079-a8ce-a697ada69c7f\") " Nov 03 15:13:09.136534 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.136512 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a0fb03-c422-430a-aa4b-6226262bcb73-util\" (UniqueName: \"kubernetes.io/empty-dir/03a0fb03-c422-430a-aa4b-6226262bcb73-util\") pod \"03a0fb03-c422-430a-aa4b-6226262bcb73\" (UID: \"03a0fb03-c422-430a-aa4b-6226262bcb73\") " Nov 03 15:13:09.136589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.136556 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af18a4e-7c82-49f3-9edb-8de433790ea3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9af18a4e-7c82-49f3-9edb-8de433790ea3-bundle\") pod \"9af18a4e-7c82-49f3-9edb-8de433790ea3\" (UID: \"9af18a4e-7c82-49f3-9edb-8de433790ea3\") " Nov 03 15:13:09.136632 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.136617 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8340b3-582d-4d77-98af-c21b5cbaeeac-kube-api-access-bmmp7\" (UniqueName: \"kubernetes.io/projected/2c8340b3-582d-4d77-98af-c21b5cbaeeac-kube-api-access-bmmp7\") pod \"2c8340b3-582d-4d77-98af-c21b5cbaeeac\" (UID: \"2c8340b3-582d-4d77-98af-c21b5cbaeeac\") " Nov 03 15:13:09.136683 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.136670 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5deaed3c-667f-4079-a8ce-a697ada69c7f-util\" (UniqueName: \"kubernetes.io/empty-dir/5deaed3c-667f-4079-a8ce-a697ada69c7f-util\") pod \"5deaed3c-667f-4079-a8ce-a697ada69c7f\" (UID: \"5deaed3c-667f-4079-a8ce-a697ada69c7f\") " Nov 03 15:13:09.136784 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.136769 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a0fb03-c422-430a-aa4b-6226262bcb73-bundle\" (UniqueName: \"kubernetes.io/empty-dir/03a0fb03-c422-430a-aa4b-6226262bcb73-bundle\") pod \"03a0fb03-c422-430a-aa4b-6226262bcb73\" (UID: \"03a0fb03-c422-430a-aa4b-6226262bcb73\") " Nov 03 15:13:09.137115 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.137098 4290 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/998236af-9ffc-4b16-9a61-5188945ff1d0-util\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.137145 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.137124 4290 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/778c932f-5f44-476a-b6ca-dd489022e1e6-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.137145 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.137138 4290 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/778c932f-5f44-476a-b6ca-dd489022e1e6-util\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.137202 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.137151 4290 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/998236af-9ffc-4b16-9a61-5188945ff1d0-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.137202 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.137165 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kml4m\" (UniqueName: \"kubernetes.io/projected/778c932f-5f44-476a-b6ca-dd489022e1e6-kube-api-access-kml4m\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.137202 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.137181 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9zqf\" (UniqueName: \"kubernetes.io/projected/998236af-9ffc-4b16-9a61-5188945ff1d0-kube-api-access-g9zqf\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.137202 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.137195 4290 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f351bf29-892f-4c6f-9c1a-b284adb73e58-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.137309 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.137208 4290 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f351bf29-892f-4c6f-9c1a-b284adb73e58-util\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.137309 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.137221 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt682\" (UniqueName: \"kubernetes.io/projected/f351bf29-892f-4c6f-9c1a-b284adb73e58-kube-api-access-vt682\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.138493 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.137681 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a0fb03-c422-430a-aa4b-6226262bcb73-bundle" pod "03a0fb03-c422-430a-aa4b-6226262bcb73" (UID: "03a0fb03-c422-430a-aa4b-6226262bcb73"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:09.138638 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.138609 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8340b3-582d-4d77-98af-c21b5cbaeeac-bundle" pod "2c8340b3-582d-4d77-98af-c21b5cbaeeac" (UID: "2c8340b3-582d-4d77-98af-c21b5cbaeeac"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:09.139030 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.138986 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af18a4e-7c82-49f3-9edb-8de433790ea3-bundle" pod "9af18a4e-7c82-49f3-9edb-8de433790ea3" (UID: "9af18a4e-7c82-49f3-9edb-8de433790ea3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:09.139613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.139592 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5deaed3c-667f-4079-a8ce-a697ada69c7f-bundle" pod "5deaed3c-667f-4079-a8ce-a697ada69c7f" (UID: "5deaed3c-667f-4079-a8ce-a697ada69c7f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:09.143751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.140205 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af18a4e-7c82-49f3-9edb-8de433790ea3-kube-api-access-sxrtd" pod "9af18a4e-7c82-49f3-9edb-8de433790ea3" (UID: "9af18a4e-7c82-49f3-9edb-8de433790ea3"). InnerVolumeSpecName "kube-api-access-sxrtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:13:09.148045 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.148020 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af18a4e-7c82-49f3-9edb-8de433790ea3-util" pod "9af18a4e-7c82-49f3-9edb-8de433790ea3" (UID: "9af18a4e-7c82-49f3-9edb-8de433790ea3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:09.149872 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.149841 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a0fb03-c422-430a-aa4b-6226262bcb73-util" pod "03a0fb03-c422-430a-aa4b-6226262bcb73" (UID: "03a0fb03-c422-430a-aa4b-6226262bcb73"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:09.154155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.154119 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a0fb03-c422-430a-aa4b-6226262bcb73-kube-api-access-46p8b" pod "03a0fb03-c422-430a-aa4b-6226262bcb73" (UID: "03a0fb03-c422-430a-aa4b-6226262bcb73"). InnerVolumeSpecName "kube-api-access-46p8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:13:09.154542 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.154516 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8340b3-582d-4d77-98af-c21b5cbaeeac-kube-api-access-bmmp7" pod "2c8340b3-582d-4d77-98af-c21b5cbaeeac" (UID: "2c8340b3-582d-4d77-98af-c21b5cbaeeac"). InnerVolumeSpecName "kube-api-access-bmmp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:13:09.156320 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.156291 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5deaed3c-667f-4079-a8ce-a697ada69c7f-util" pod "5deaed3c-667f-4079-a8ce-a697ada69c7f" (UID: "5deaed3c-667f-4079-a8ce-a697ada69c7f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:09.158172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.158146 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8340b3-582d-4d77-98af-c21b5cbaeeac-util" pod "2c8340b3-582d-4d77-98af-c21b5cbaeeac" (UID: "2c8340b3-582d-4d77-98af-c21b5cbaeeac"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:09.158627 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.158607 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5deaed3c-667f-4079-a8ce-a697ada69c7f-kube-api-access-nx275" pod "5deaed3c-667f-4079-a8ce-a697ada69c7f" (UID: "5deaed3c-667f-4079-a8ce-a697ada69c7f"). InnerVolumeSpecName "kube-api-access-nx275". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:13:09.237827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.237801 4290 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c8340b3-582d-4d77-98af-c21b5cbaeeac-util\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.237827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.237824 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx275\" (UniqueName: \"kubernetes.io/projected/5deaed3c-667f-4079-a8ce-a697ada69c7f-kube-api-access-nx275\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.237962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.237834 4290 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5deaed3c-667f-4079-a8ce-a697ada69c7f-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.237962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.237844 4290 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03a0fb03-c422-430a-aa4b-6226262bcb73-util\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.237962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.237852 4290 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9af18a4e-7c82-49f3-9edb-8de433790ea3-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.237962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.237860 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmmp7\" (UniqueName: \"kubernetes.io/projected/2c8340b3-582d-4d77-98af-c21b5cbaeeac-kube-api-access-bmmp7\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.237962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.237867 4290 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5deaed3c-667f-4079-a8ce-a697ada69c7f-util\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.237962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.237875 4290 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03a0fb03-c422-430a-aa4b-6226262bcb73-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.237962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.237882 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxrtd\" (UniqueName: \"kubernetes.io/projected/9af18a4e-7c82-49f3-9edb-8de433790ea3-kube-api-access-sxrtd\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.237962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.237890 4290 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9af18a4e-7c82-49f3-9edb-8de433790ea3-util\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.237962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.237898 4290 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c8340b3-582d-4d77-98af-c21b5cbaeeac-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.237962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.237906 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46p8b\" (UniqueName: \"kubernetes.io/projected/03a0fb03-c422-430a-aa4b-6226262bcb73-kube-api-access-46p8b\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:09.581099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.581075 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp"] Nov 03 15:13:09.582874 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:13:09.582847 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ecd436a_2f5e_4119_b25b_66902c0140b8.slice/crio-6400fe332ae2ce40054ba3867a4940acf4cd31e8682cb68485e881076c6356e3 WatchSource:0}: Error finding container 6400fe332ae2ce40054ba3867a4940acf4cd31e8682cb68485e881076c6356e3: Status 404 returned error can't find the container with id 6400fe332ae2ce40054ba3867a4940acf4cd31e8682cb68485e881076c6356e3 Nov 03 15:13:09.664989 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.664959 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" event={"ID":"2c8340b3-582d-4d77-98af-c21b5cbaeeac","Type":"ContainerDied","Data":"bf56b7dc80c5306649c66825dd32add27d7c693a13dca572853a8b4204d94a23"} Nov 03 15:13:09.665425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.665008 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf56b7dc80c5306649c66825dd32add27d7c693a13dca572853a8b4204d94a23" Nov 03 15:13:09.665425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.664971 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66vkjv" Nov 03 15:13:09.666942 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.666917 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" event={"ID":"998236af-9ffc-4b16-9a61-5188945ff1d0","Type":"ContainerDied","Data":"294cda7c18d8aa8cccba6f7bbec16f70b4ce410dee51cf5cc647ee95b40d0292"} Nov 03 15:13:09.667069 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.666953 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="294cda7c18d8aa8cccba6f7bbec16f70b4ce410dee51cf5cc647ee95b40d0292" Nov 03 15:13:09.667069 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.666926 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8k86r" Nov 03 15:13:09.668441 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.668420 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:13:09.669180 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.669149 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" event={"ID":"03a0fb03-c422-430a-aa4b-6226262bcb73","Type":"ContainerDied","Data":"99490aebf342de636c678c416c3bca566777346ca2003c19edd036b029e3150f"} Nov 03 15:13:09.669180 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.669163 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8" Nov 03 15:13:09.669330 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.669192 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99490aebf342de636c678c416c3bca566777346ca2003c19edd036b029e3150f" Nov 03 15:13:09.670758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.670736 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:13:09.671689 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.671674 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" Nov 03 15:13:09.671814 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.671675 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k" event={"ID":"f351bf29-892f-4c6f-9c1a-b284adb73e58","Type":"ContainerDied","Data":"3960ab939e4208fead97157c7de665aa9e742a0ac7e83a1018b4c45438ee20fb"} Nov 03 15:13:09.671814 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.671749 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3960ab939e4208fead97157c7de665aa9e742a0ac7e83a1018b4c45438ee20fb" Nov 03 15:13:09.673979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.673730 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" Nov 03 15:13:09.673979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.673729 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78fkh8" event={"ID":"9af18a4e-7c82-49f3-9edb-8de433790ea3","Type":"ContainerDied","Data":"32af132b174262d288fea8e062ef7d2fd79426cb3c725838942ee39b3e3a5ca2"} Nov 03 15:13:09.673979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.673848 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32af132b174262d288fea8e062ef7d2fd79426cb3c725838942ee39b3e3a5ca2" Nov 03 15:13:09.687147 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.687118 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" event={"ID":"778c932f-5f44-476a-b6ca-dd489022e1e6","Type":"ContainerDied","Data":"fdce56443558fcd92019ce55d2913f11c48751b209a5b519dfbc83b4b64f2d95"} Nov 03 15:13:09.687379 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.687357 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdce56443558fcd92019ce55d2913f11c48751b209a5b519dfbc83b4b64f2d95" Nov 03 15:13:09.687480 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.687176 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avjd5q" Nov 03 15:13:09.689068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.689045 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" event={"ID":"9ecd436a-2f5e-4119-b25b-66902c0140b8","Type":"ContainerStarted","Data":"6400fe332ae2ce40054ba3867a4940acf4cd31e8682cb68485e881076c6356e3"} Nov 03 15:13:09.691401 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.691381 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" event={"ID":"5deaed3c-667f-4079-a8ce-a697ada69c7f","Type":"ContainerDied","Data":"ab325eb4369d58612df2e5b06b7fab78d080d4edb047e5ef4956cca46c38bd91"} Nov 03 15:13:09.691496 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.691413 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab325eb4369d58612df2e5b06b7fab78d080d4edb047e5ef4956cca46c38bd91" Nov 03 15:13:09.691496 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:09.691427 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm9xg9" Nov 03 15:13:12.674205 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.674177 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:13:12.674848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.674822 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:13:12.677549 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.677530 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:13:12.678410 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.678392 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:13:12.933203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933129 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-gc7bt"] Nov 03 15:13:12.933545 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933532 4290 cpu_manager.go:414] "containerMap: removing container" podUID="03a0fb03-c422-430a-aa4b-6226262bcb73" containerName="util" Nov 03 15:13:12.933581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933549 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a0fb03-c422-430a-aa4b-6226262bcb73" containerName="util" Nov 03 15:13:12.933581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933558 4290 cpu_manager.go:414] "containerMap: removing container" podUID="998236af-9ffc-4b16-9a61-5188945ff1d0" containerName="pull" Nov 03 15:13:12.933581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933563 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998236af-9ffc-4b16-9a61-5188945ff1d0" containerName="pull" Nov 03 15:13:12.933581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933572 4290 cpu_manager.go:414] "containerMap: removing container" podUID="03a0fb03-c422-430a-aa4b-6226262bcb73" containerName="pull" Nov 03 15:13:12.933581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933576 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a0fb03-c422-430a-aa4b-6226262bcb73" containerName="pull" Nov 03 15:13:12.933581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933582 4290 cpu_manager.go:414] "containerMap: removing container" podUID="2c8340b3-582d-4d77-98af-c21b5cbaeeac" containerName="extract" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933587 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8340b3-582d-4d77-98af-c21b5cbaeeac" containerName="extract" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933605 4290 cpu_manager.go:414] "containerMap: removing container" podUID="5deaed3c-667f-4079-a8ce-a697ada69c7f" containerName="util" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933615 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5deaed3c-667f-4079-a8ce-a697ada69c7f" containerName="util" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933629 4290 cpu_manager.go:414] "containerMap: removing container" podUID="9af18a4e-7c82-49f3-9edb-8de433790ea3" containerName="pull" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933634 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af18a4e-7c82-49f3-9edb-8de433790ea3" containerName="pull" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933643 4290 cpu_manager.go:414] "containerMap: removing container" podUID="998236af-9ffc-4b16-9a61-5188945ff1d0" containerName="extract" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933647 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998236af-9ffc-4b16-9a61-5188945ff1d0" containerName="extract" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933655 4290 cpu_manager.go:414] "containerMap: removing container" podUID="2c8340b3-582d-4d77-98af-c21b5cbaeeac" containerName="util" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933659 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8340b3-582d-4d77-98af-c21b5cbaeeac" containerName="util" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933666 4290 cpu_manager.go:414] "containerMap: removing container" podUID="9af18a4e-7c82-49f3-9edb-8de433790ea3" containerName="util" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933670 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af18a4e-7c82-49f3-9edb-8de433790ea3" containerName="util" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933676 4290 cpu_manager.go:414] "containerMap: removing container" podUID="f351bf29-892f-4c6f-9c1a-b284adb73e58" containerName="pull" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933681 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f351bf29-892f-4c6f-9c1a-b284adb73e58" containerName="pull" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933694 4290 cpu_manager.go:414] "containerMap: removing container" podUID="5deaed3c-667f-4079-a8ce-a697ada69c7f" containerName="pull" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933724 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5deaed3c-667f-4079-a8ce-a697ada69c7f" containerName="pull" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933731 4290 cpu_manager.go:414] "containerMap: removing container" podUID="2c8340b3-582d-4d77-98af-c21b5cbaeeac" containerName="pull" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933735 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8340b3-582d-4d77-98af-c21b5cbaeeac" containerName="pull" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933740 4290 cpu_manager.go:414] "containerMap: removing container" podUID="778c932f-5f44-476a-b6ca-dd489022e1e6" containerName="extract" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933744 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778c932f-5f44-476a-b6ca-dd489022e1e6" containerName="extract" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933751 4290 cpu_manager.go:414] "containerMap: removing container" podUID="f351bf29-892f-4c6f-9c1a-b284adb73e58" containerName="util" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933756 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f351bf29-892f-4c6f-9c1a-b284adb73e58" containerName="util" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933763 4290 cpu_manager.go:414] "containerMap: removing container" podUID="778c932f-5f44-476a-b6ca-dd489022e1e6" containerName="util" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933769 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778c932f-5f44-476a-b6ca-dd489022e1e6" containerName="util" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933778 4290 cpu_manager.go:414] "containerMap: removing container" podUID="03a0fb03-c422-430a-aa4b-6226262bcb73" containerName="extract" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933785 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a0fb03-c422-430a-aa4b-6226262bcb73" containerName="extract" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933794 4290 cpu_manager.go:414] "containerMap: removing container" podUID="9af18a4e-7c82-49f3-9edb-8de433790ea3" containerName="extract" Nov 03 15:13:12.933791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933798 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af18a4e-7c82-49f3-9edb-8de433790ea3" containerName="extract" Nov 03 15:13:12.934482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933806 4290 cpu_manager.go:414] "containerMap: removing container" podUID="5deaed3c-667f-4079-a8ce-a697ada69c7f" containerName="extract" Nov 03 15:13:12.934482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933811 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5deaed3c-667f-4079-a8ce-a697ada69c7f" containerName="extract" Nov 03 15:13:12.934482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933820 4290 cpu_manager.go:414] "containerMap: removing container" podUID="998236af-9ffc-4b16-9a61-5188945ff1d0" containerName="util" Nov 03 15:13:12.934482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933824 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998236af-9ffc-4b16-9a61-5188945ff1d0" containerName="util" Nov 03 15:13:12.934482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933829 4290 cpu_manager.go:414] "containerMap: removing container" podUID="778c932f-5f44-476a-b6ca-dd489022e1e6" containerName="pull" Nov 03 15:13:12.934482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933834 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778c932f-5f44-476a-b6ca-dd489022e1e6" containerName="pull" Nov 03 15:13:12.934482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933838 4290 cpu_manager.go:414] "containerMap: removing container" podUID="f351bf29-892f-4c6f-9c1a-b284adb73e58" containerName="extract" Nov 03 15:13:12.934482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933842 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f351bf29-892f-4c6f-9c1a-b284adb73e58" containerName="extract" Nov 03 15:13:12.934482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933905 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8340b3-582d-4d77-98af-c21b5cbaeeac" containerName="extract" Nov 03 15:13:12.934482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933916 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="998236af-9ffc-4b16-9a61-5188945ff1d0" containerName="extract" Nov 03 15:13:12.934482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933923 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a0fb03-c422-430a-aa4b-6226262bcb73" containerName="extract" Nov 03 15:13:12.934482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933930 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="778c932f-5f44-476a-b6ca-dd489022e1e6" containerName="extract" Nov 03 15:13:12.934482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933939 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af18a4e-7c82-49f3-9edb-8de433790ea3" containerName="extract" Nov 03 15:13:12.934482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933946 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="5deaed3c-667f-4079-a8ce-a697ada69c7f" containerName="extract" Nov 03 15:13:12.934482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.933950 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="f351bf29-892f-4c6f-9c1a-b284adb73e58" containerName="extract" Nov 03 15:13:12.996540 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.996513 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gc7bt" Nov 03 15:13:12.998123 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:12.998095 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-gc7bt"] Nov 03 15:13:13.050809 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:13.050781 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-m6hlx\"" Nov 03 15:13:13.078886 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:13.078862 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njn9j\" (UniqueName: \"kubernetes.io/projected/d1f11052-977d-4abe-a71a-17bb67335464-kube-api-access-njn9j\") pod \"jobset-operator-674cd59dc5-gc7bt\" (UID: \"d1f11052-977d-4abe-a71a-17bb67335464\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gc7bt" Nov 03 15:13:13.078963 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:13.078915 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1f11052-977d-4abe-a71a-17bb67335464-tmp\") pod \"jobset-operator-674cd59dc5-gc7bt\" (UID: \"d1f11052-977d-4abe-a71a-17bb67335464\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gc7bt" Nov 03 15:13:13.100172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:13.100151 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 15:13:13.100172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:13.100166 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 15:13:13.179368 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:13.179347 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njn9j\" (UniqueName: \"kubernetes.io/projected/d1f11052-977d-4abe-a71a-17bb67335464-kube-api-access-njn9j\") pod \"jobset-operator-674cd59dc5-gc7bt\" (UID: \"d1f11052-977d-4abe-a71a-17bb67335464\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gc7bt" Nov 03 15:13:13.179472 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:13.179398 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1f11052-977d-4abe-a71a-17bb67335464-tmp\") pod \"jobset-operator-674cd59dc5-gc7bt\" (UID: \"d1f11052-977d-4abe-a71a-17bb67335464\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gc7bt" Nov 03 15:13:13.179766 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:13.179749 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1f11052-977d-4abe-a71a-17bb67335464-tmp\") pod \"jobset-operator-674cd59dc5-gc7bt\" (UID: \"d1f11052-977d-4abe-a71a-17bb67335464\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gc7bt" Nov 03 15:13:13.243604 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:13.243557 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njn9j\" (UniqueName: \"kubernetes.io/projected/d1f11052-977d-4abe-a71a-17bb67335464-kube-api-access-njn9j\") pod \"jobset-operator-674cd59dc5-gc7bt\" (UID: \"d1f11052-977d-4abe-a71a-17bb67335464\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gc7bt" Nov 03 15:13:13.346341 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:13.346318 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gc7bt" Nov 03 15:13:13.733095 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:13.733057 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" event={"ID":"9ecd436a-2f5e-4119-b25b-66902c0140b8","Type":"ContainerStarted","Data":"d570f4d56ba54b4b737ea407a39ff65363a88fb4b0727b443bc410f273fb1237"} Nov 03 15:13:13.860846 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:13.860778 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-gc7bt"] Nov 03 15:13:13.863094 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:13:13.863066 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1f11052_977d_4abe_a71a_17bb67335464.slice/crio-74a80034c133371d42e377c6e2fe3b44dc0ae62590a4d70476ae50baab1f8e16 WatchSource:0}: Error finding container 74a80034c133371d42e377c6e2fe3b44dc0ae62590a4d70476ae50baab1f8e16: Status 404 returned error can't find the container with id 74a80034c133371d42e377c6e2fe3b44dc0ae62590a4d70476ae50baab1f8e16 Nov 03 15:13:14.740067 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:14.740031 4290 generic.go:356] "Generic (PLEG): container finished" podID="9ecd436a-2f5e-4119-b25b-66902c0140b8" containerID="d570f4d56ba54b4b737ea407a39ff65363a88fb4b0727b443bc410f273fb1237" exitCode=0 Nov 03 15:13:14.740488 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:14.740122 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" event={"ID":"9ecd436a-2f5e-4119-b25b-66902c0140b8","Type":"ContainerDied","Data":"d570f4d56ba54b4b737ea407a39ff65363a88fb4b0727b443bc410f273fb1237"} Nov 03 15:13:14.740735 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:14.740718 4290 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:13:14.741662 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:14.741629 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gc7bt" event={"ID":"d1f11052-977d-4abe-a71a-17bb67335464","Type":"ContainerStarted","Data":"74a80034c133371d42e377c6e2fe3b44dc0ae62590a4d70476ae50baab1f8e16"} Nov 03 15:13:15.749343 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:15.749303 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" event={"ID":"9ecd436a-2f5e-4119-b25b-66902c0140b8","Type":"ContainerStarted","Data":"e8252c59ad4f743a6c79c4b80da92101de8fb4a9c2c7dc3daa884f5afbef5055"} Nov 03 15:13:15.749780 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:15.749462 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:15.947034 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:15.946955 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" podStartSLOduration=2.799744158 podStartE2EDuration="7.946935811s" totalImagesPullingTime="4.685666654s" totalInitContainerRuntime="461.524999ms" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:15.946770742 +0000 UTC m=+303.377728551" watchObservedRunningTime="2026-11-03 15:13:15.946935811 +0000 UTC m=+303.377893619" Nov 03 15:13:17.057257 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.057184 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv"] Nov 03 15:13:17.059733 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.059694 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh"] Nov 03 15:13:17.059836 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.059820 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" Nov 03 15:13:17.061916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.061896 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" Nov 03 15:13:17.110681 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.110656 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14729f5a-6560-44de-9d97-95101db87301-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv\" (UID: \"14729f5a-6560-44de-9d97-95101db87301\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" Nov 03 15:13:17.110833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.110685 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10b03ad1-e066-4092-9a69-b5ea770abfc4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh\" (UID: \"10b03ad1-e066-4092-9a69-b5ea770abfc4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" Nov 03 15:13:17.110833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.110793 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14729f5a-6560-44de-9d97-95101db87301-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv\" (UID: \"14729f5a-6560-44de-9d97-95101db87301\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" Nov 03 15:13:17.110952 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.110933 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqpn4\" (UniqueName: \"kubernetes.io/projected/14729f5a-6560-44de-9d97-95101db87301-kube-api-access-fqpn4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv\" (UID: \"14729f5a-6560-44de-9d97-95101db87301\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" Nov 03 15:13:17.110991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.110970 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10b03ad1-e066-4092-9a69-b5ea770abfc4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh\" (UID: \"10b03ad1-e066-4092-9a69-b5ea770abfc4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" Nov 03 15:13:17.110991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.110987 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r424\" (UniqueName: \"kubernetes.io/projected/10b03ad1-e066-4092-9a69-b5ea770abfc4-kube-api-access-6r424\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh\" (UID: \"10b03ad1-e066-4092-9a69-b5ea770abfc4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" Nov 03 15:13:17.119762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.119733 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-lg7t2\"" Nov 03 15:13:17.119865 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.119837 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 15:13:17.119918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.119890 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-j4kj6\"" Nov 03 15:13:17.169466 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.169443 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 15:13:17.171962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.171944 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv"] Nov 03 15:13:17.173956 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.173936 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh"] Nov 03 15:13:17.211570 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.211541 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqpn4\" (UniqueName: \"kubernetes.io/projected/14729f5a-6560-44de-9d97-95101db87301-kube-api-access-fqpn4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv\" (UID: \"14729f5a-6560-44de-9d97-95101db87301\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" Nov 03 15:13:17.211734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.211577 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10b03ad1-e066-4092-9a69-b5ea770abfc4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh\" (UID: \"10b03ad1-e066-4092-9a69-b5ea770abfc4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" Nov 03 15:13:17.211734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.211594 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r424\" (UniqueName: \"kubernetes.io/projected/10b03ad1-e066-4092-9a69-b5ea770abfc4-kube-api-access-6r424\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh\" (UID: \"10b03ad1-e066-4092-9a69-b5ea770abfc4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" Nov 03 15:13:17.211734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.211624 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14729f5a-6560-44de-9d97-95101db87301-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv\" (UID: \"14729f5a-6560-44de-9d97-95101db87301\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" Nov 03 15:13:17.211734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.211645 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10b03ad1-e066-4092-9a69-b5ea770abfc4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh\" (UID: \"10b03ad1-e066-4092-9a69-b5ea770abfc4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" Nov 03 15:13:17.211734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.211695 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14729f5a-6560-44de-9d97-95101db87301-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv\" (UID: \"14729f5a-6560-44de-9d97-95101db87301\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" Nov 03 15:13:17.212034 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.212012 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14729f5a-6560-44de-9d97-95101db87301-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv\" (UID: \"14729f5a-6560-44de-9d97-95101db87301\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" Nov 03 15:13:17.212097 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.212043 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10b03ad1-e066-4092-9a69-b5ea770abfc4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh\" (UID: \"10b03ad1-e066-4092-9a69-b5ea770abfc4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" Nov 03 15:13:17.212097 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.212064 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14729f5a-6560-44de-9d97-95101db87301-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv\" (UID: \"14729f5a-6560-44de-9d97-95101db87301\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" Nov 03 15:13:17.212197 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.212089 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10b03ad1-e066-4092-9a69-b5ea770abfc4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh\" (UID: \"10b03ad1-e066-4092-9a69-b5ea770abfc4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" Nov 03 15:13:17.273044 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.273018 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r424\" (UniqueName: \"kubernetes.io/projected/10b03ad1-e066-4092-9a69-b5ea770abfc4-kube-api-access-6r424\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh\" (UID: \"10b03ad1-e066-4092-9a69-b5ea770abfc4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" Nov 03 15:13:17.273161 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.273100 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqpn4\" (UniqueName: \"kubernetes.io/projected/14729f5a-6560-44de-9d97-95101db87301-kube-api-access-fqpn4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv\" (UID: \"14729f5a-6560-44de-9d97-95101db87301\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" Nov 03 15:13:17.374138 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.374113 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" Nov 03 15:13:17.378387 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.378367 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" Nov 03 15:13:17.764613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.764533 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gc7bt" event={"ID":"d1f11052-977d-4abe-a71a-17bb67335464","Type":"ContainerStarted","Data":"87e6076cb662bfa0653277f5fe4f4a935a1160e72994fc9bd745220e82532f5e"} Nov 03 15:13:17.815405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.815380 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh"] Nov 03 15:13:17.818076 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:13:17.818035 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b03ad1_e066_4092_9a69_b5ea770abfc4.slice/crio-b907a179df11403f19687840dc6b191f10744264c77fb6348b6af44157614481 WatchSource:0}: Error finding container b907a179df11403f19687840dc6b191f10744264c77fb6348b6af44157614481: Status 404 returned error can't find the container with id b907a179df11403f19687840dc6b191f10744264c77fb6348b6af44157614481 Nov 03 15:13:17.820113 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.820091 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv"] Nov 03 15:13:17.822015 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:13:17.821992 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14729f5a_6560_44de_9d97_95101db87301.slice/crio-1673913aaef35ba7e4b0dc97d4d1000ca95dba4c1103d572b202d77aff5fb0ae WatchSource:0}: Error finding container 1673913aaef35ba7e4b0dc97d4d1000ca95dba4c1103d572b202d77aff5fb0ae: Status 404 returned error can't find the container with id 1673913aaef35ba7e4b0dc97d4d1000ca95dba4c1103d572b202d77aff5fb0ae Nov 03 15:13:17.887777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:17.887721 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gc7bt" podStartSLOduration=3.060985737 podStartE2EDuration="5.887688796s" totalImagesPullingTime="2.826703059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:17.88686327 +0000 UTC m=+305.317821081" watchObservedRunningTime="2026-11-03 15:13:17.887688796 +0000 UTC m=+305.318646602" Nov 03 15:13:18.771086 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:18.771050 4290 generic.go:356] "Generic (PLEG): container finished" podID="10b03ad1-e066-4092-9a69-b5ea770abfc4" containerID="f892cf7bccc6cb80f0a6307aef7a04e9a71b55656daff94460c537cd24949229" exitCode=0 Nov 03 15:13:18.771527 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:18.771140 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" event={"ID":"10b03ad1-e066-4092-9a69-b5ea770abfc4","Type":"ContainerDied","Data":"f892cf7bccc6cb80f0a6307aef7a04e9a71b55656daff94460c537cd24949229"} Nov 03 15:13:18.771527 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:18.771190 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" event={"ID":"10b03ad1-e066-4092-9a69-b5ea770abfc4","Type":"ContainerStarted","Data":"b907a179df11403f19687840dc6b191f10744264c77fb6348b6af44157614481"} Nov 03 15:13:18.772956 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:18.772934 4290 generic.go:356] "Generic (PLEG): container finished" podID="14729f5a-6560-44de-9d97-95101db87301" containerID="af4b2a2965c741c6ec8806c55ce4d64723d96b32962bd9b9a60f22f13bd05e03" exitCode=0 Nov 03 15:13:18.773535 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:18.773512 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" event={"ID":"14729f5a-6560-44de-9d97-95101db87301","Type":"ContainerDied","Data":"af4b2a2965c741c6ec8806c55ce4d64723d96b32962bd9b9a60f22f13bd05e03"} Nov 03 15:13:18.773648 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:18.773546 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" event={"ID":"14729f5a-6560-44de-9d97-95101db87301","Type":"ContainerStarted","Data":"1673913aaef35ba7e4b0dc97d4d1000ca95dba4c1103d572b202d77aff5fb0ae"} Nov 03 15:13:19.783182 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:19.783144 4290 generic.go:356] "Generic (PLEG): container finished" podID="10b03ad1-e066-4092-9a69-b5ea770abfc4" containerID="53efcdfdb562d37393e163c7ab2819f36e792cd9a6b28dfc81a5fe18d9c9a2b7" exitCode=0 Nov 03 15:13:19.783617 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:19.783232 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" event={"ID":"10b03ad1-e066-4092-9a69-b5ea770abfc4","Type":"ContainerDied","Data":"53efcdfdb562d37393e163c7ab2819f36e792cd9a6b28dfc81a5fe18d9c9a2b7"} Nov 03 15:13:19.785293 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:19.785270 4290 generic.go:356] "Generic (PLEG): container finished" podID="14729f5a-6560-44de-9d97-95101db87301" containerID="80651fc351ebc804a265a49e5ecd464e922758d289314c041e9fa21bd0c9b767" exitCode=0 Nov 03 15:13:19.785417 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:19.785306 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" event={"ID":"14729f5a-6560-44de-9d97-95101db87301","Type":"ContainerDied","Data":"80651fc351ebc804a265a49e5ecd464e922758d289314c041e9fa21bd0c9b767"} Nov 03 15:13:20.792683 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:20.792652 4290 generic.go:356] "Generic (PLEG): container finished" podID="14729f5a-6560-44de-9d97-95101db87301" containerID="1b526aea74ff9ef647e29372db9ae19648d42117771458d1110949d78555713e" exitCode=0 Nov 03 15:13:20.793115 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:20.792740 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" event={"ID":"14729f5a-6560-44de-9d97-95101db87301","Type":"ContainerDied","Data":"1b526aea74ff9ef647e29372db9ae19648d42117771458d1110949d78555713e"} Nov 03 15:13:20.795104 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:20.795080 4290 generic.go:356] "Generic (PLEG): container finished" podID="10b03ad1-e066-4092-9a69-b5ea770abfc4" containerID="341b8706e112da96ff8561040e60456ad9fc312aebaa5f4ca41ad68866922120" exitCode=0 Nov 03 15:13:20.795225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:20.795110 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" event={"ID":"10b03ad1-e066-4092-9a69-b5ea770abfc4","Type":"ContainerDied","Data":"341b8706e112da96ff8561040e60456ad9fc312aebaa5f4ca41ad68866922120"} Nov 03 15:13:21.929625 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:21.929594 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" Nov 03 15:13:21.952359 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:21.952335 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b03ad1-e066-4092-9a69-b5ea770abfc4-util\" (UniqueName: \"kubernetes.io/empty-dir/10b03ad1-e066-4092-9a69-b5ea770abfc4-util\") pod \"10b03ad1-e066-4092-9a69-b5ea770abfc4\" (UID: \"10b03ad1-e066-4092-9a69-b5ea770abfc4\") " Nov 03 15:13:21.952462 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:21.952401 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b03ad1-e066-4092-9a69-b5ea770abfc4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/10b03ad1-e066-4092-9a69-b5ea770abfc4-bundle\") pod \"10b03ad1-e066-4092-9a69-b5ea770abfc4\" (UID: \"10b03ad1-e066-4092-9a69-b5ea770abfc4\") " Nov 03 15:13:21.952529 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:21.952464 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b03ad1-e066-4092-9a69-b5ea770abfc4-kube-api-access-6r424\" (UniqueName: \"kubernetes.io/projected/10b03ad1-e066-4092-9a69-b5ea770abfc4-kube-api-access-6r424\") pod \"10b03ad1-e066-4092-9a69-b5ea770abfc4\" (UID: \"10b03ad1-e066-4092-9a69-b5ea770abfc4\") " Nov 03 15:13:21.952938 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:21.952895 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b03ad1-e066-4092-9a69-b5ea770abfc4-bundle" pod "10b03ad1-e066-4092-9a69-b5ea770abfc4" (UID: "10b03ad1-e066-4092-9a69-b5ea770abfc4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:21.955030 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:21.955007 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b03ad1-e066-4092-9a69-b5ea770abfc4-kube-api-access-6r424" pod "10b03ad1-e066-4092-9a69-b5ea770abfc4" (UID: "10b03ad1-e066-4092-9a69-b5ea770abfc4"). InnerVolumeSpecName "kube-api-access-6r424". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:13:21.955624 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:21.955608 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" Nov 03 15:13:21.959230 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:21.959210 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b03ad1-e066-4092-9a69-b5ea770abfc4-util" pod "10b03ad1-e066-4092-9a69-b5ea770abfc4" (UID: "10b03ad1-e066-4092-9a69-b5ea770abfc4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:22.053249 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.053184 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14729f5a-6560-44de-9d97-95101db87301-bundle\" (UniqueName: \"kubernetes.io/empty-dir/14729f5a-6560-44de-9d97-95101db87301-bundle\") pod \"14729f5a-6560-44de-9d97-95101db87301\" (UID: \"14729f5a-6560-44de-9d97-95101db87301\") " Nov 03 15:13:22.053249 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.053222 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14729f5a-6560-44de-9d97-95101db87301-util\" (UniqueName: \"kubernetes.io/empty-dir/14729f5a-6560-44de-9d97-95101db87301-util\") pod \"14729f5a-6560-44de-9d97-95101db87301\" (UID: \"14729f5a-6560-44de-9d97-95101db87301\") " Nov 03 15:13:22.053433 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.053280 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14729f5a-6560-44de-9d97-95101db87301-kube-api-access-fqpn4\" (UniqueName: \"kubernetes.io/projected/14729f5a-6560-44de-9d97-95101db87301-kube-api-access-fqpn4\") pod \"14729f5a-6560-44de-9d97-95101db87301\" (UID: \"14729f5a-6560-44de-9d97-95101db87301\") " Nov 03 15:13:22.053514 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.053496 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r424\" (UniqueName: \"kubernetes.io/projected/10b03ad1-e066-4092-9a69-b5ea770abfc4-kube-api-access-6r424\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:22.053581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.053516 4290 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10b03ad1-e066-4092-9a69-b5ea770abfc4-util\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:22.053581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.053525 4290 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10b03ad1-e066-4092-9a69-b5ea770abfc4-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:22.053685 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.053607 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14729f5a-6560-44de-9d97-95101db87301-bundle" pod "14729f5a-6560-44de-9d97-95101db87301" (UID: "14729f5a-6560-44de-9d97-95101db87301"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:22.055509 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.055452 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14729f5a-6560-44de-9d97-95101db87301-kube-api-access-fqpn4" pod "14729f5a-6560-44de-9d97-95101db87301" (UID: "14729f5a-6560-44de-9d97-95101db87301"). InnerVolumeSpecName "kube-api-access-fqpn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:13:22.060830 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.060792 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14729f5a-6560-44de-9d97-95101db87301-util" pod "14729f5a-6560-44de-9d97-95101db87301" (UID: "14729f5a-6560-44de-9d97-95101db87301"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:13:22.154247 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.154222 4290 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14729f5a-6560-44de-9d97-95101db87301-bundle\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:22.154247 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.154244 4290 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14729f5a-6560-44de-9d97-95101db87301-util\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:22.154247 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.154254 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqpn4\" (UniqueName: \"kubernetes.io/projected/14729f5a-6560-44de-9d97-95101db87301-kube-api-access-fqpn4\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:22.807579 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.807547 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" Nov 03 15:13:22.807794 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.807542 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tz8sh" event={"ID":"10b03ad1-e066-4092-9a69-b5ea770abfc4","Type":"ContainerDied","Data":"b907a179df11403f19687840dc6b191f10744264c77fb6348b6af44157614481"} Nov 03 15:13:22.807794 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.807679 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b907a179df11403f19687840dc6b191f10744264c77fb6348b6af44157614481" Nov 03 15:13:22.809669 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.809588 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" event={"ID":"14729f5a-6560-44de-9d97-95101db87301","Type":"ContainerDied","Data":"1673913aaef35ba7e4b0dc97d4d1000ca95dba4c1103d572b202d77aff5fb0ae"} Nov 03 15:13:22.809669 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.809640 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1673913aaef35ba7e4b0dc97d4d1000ca95dba4c1103d572b202d77aff5fb0ae" Nov 03 15:13:22.809897 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:22.809672 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zfqcv" Nov 03 15:13:23.662196 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.662160 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8rngp"] Nov 03 15:13:23.662691 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.662669 4290 cpu_manager.go:414] "containerMap: removing container" podUID="14729f5a-6560-44de-9d97-95101db87301" containerName="pull" Nov 03 15:13:23.662691 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.662695 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14729f5a-6560-44de-9d97-95101db87301" containerName="pull" Nov 03 15:13:23.662863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.662734 4290 cpu_manager.go:414] "containerMap: removing container" podUID="10b03ad1-e066-4092-9a69-b5ea770abfc4" containerName="util" Nov 03 15:13:23.662863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.662741 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b03ad1-e066-4092-9a69-b5ea770abfc4" containerName="util" Nov 03 15:13:23.662863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.662753 4290 cpu_manager.go:414] "containerMap: removing container" podUID="14729f5a-6560-44de-9d97-95101db87301" containerName="extract" Nov 03 15:13:23.662863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.662762 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14729f5a-6560-44de-9d97-95101db87301" containerName="extract" Nov 03 15:13:23.662863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.662789 4290 cpu_manager.go:414] "containerMap: removing container" podUID="10b03ad1-e066-4092-9a69-b5ea770abfc4" containerName="pull" Nov 03 15:13:23.662863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.662797 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b03ad1-e066-4092-9a69-b5ea770abfc4" containerName="pull" Nov 03 15:13:23.662863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.662824 4290 cpu_manager.go:414] "containerMap: removing container" podUID="14729f5a-6560-44de-9d97-95101db87301" containerName="util" Nov 03 15:13:23.662863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.662832 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14729f5a-6560-44de-9d97-95101db87301" containerName="util" Nov 03 15:13:23.662863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.662844 4290 cpu_manager.go:414] "containerMap: removing container" podUID="10b03ad1-e066-4092-9a69-b5ea770abfc4" containerName="extract" Nov 03 15:13:23.662863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.662851 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b03ad1-e066-4092-9a69-b5ea770abfc4" containerName="extract" Nov 03 15:13:23.663316 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.662955 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="14729f5a-6560-44de-9d97-95101db87301" containerName="extract" Nov 03 15:13:23.663316 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.662974 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b03ad1-e066-4092-9a69-b5ea770abfc4" containerName="extract" Nov 03 15:13:23.665067 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.665047 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8rngp" Nov 03 15:13:23.720542 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.720507 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 15:13:23.766624 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.766588 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e95fe44-e76f-4306-87a2-0ff1e68ca5cd-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-8rngp\" (UID: \"0e95fe44-e76f-4306-87a2-0ff1e68ca5cd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8rngp" Nov 03 15:13:23.766779 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.766762 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97s8b\" (UniqueName: \"kubernetes.io/projected/0e95fe44-e76f-4306-87a2-0ff1e68ca5cd-kube-api-access-97s8b\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-8rngp\" (UID: \"0e95fe44-e76f-4306-87a2-0ff1e68ca5cd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8rngp" Nov 03 15:13:23.770261 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.770243 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 15:13:23.770392 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.770274 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-b2bts\"" Nov 03 15:13:23.772588 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.772569 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8rngp"] Nov 03 15:13:23.867416 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.867379 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97s8b\" (UniqueName: \"kubernetes.io/projected/0e95fe44-e76f-4306-87a2-0ff1e68ca5cd-kube-api-access-97s8b\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-8rngp\" (UID: \"0e95fe44-e76f-4306-87a2-0ff1e68ca5cd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8rngp" Nov 03 15:13:23.867587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.867429 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e95fe44-e76f-4306-87a2-0ff1e68ca5cd-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-8rngp\" (UID: \"0e95fe44-e76f-4306-87a2-0ff1e68ca5cd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8rngp" Nov 03 15:13:23.867793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.867777 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e95fe44-e76f-4306-87a2-0ff1e68ca5cd-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-8rngp\" (UID: \"0e95fe44-e76f-4306-87a2-0ff1e68ca5cd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8rngp" Nov 03 15:13:23.929500 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.929420 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97s8b\" (UniqueName: \"kubernetes.io/projected/0e95fe44-e76f-4306-87a2-0ff1e68ca5cd-kube-api-access-97s8b\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-8rngp\" (UID: \"0e95fe44-e76f-4306-87a2-0ff1e68ca5cd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8rngp" Nov 03 15:13:23.975189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:23.975154 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8rngp" Nov 03 15:13:24.411344 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:24.411319 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8rngp"] Nov 03 15:13:24.413069 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:13:24.413041 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e95fe44_e76f_4306_87a2_0ff1e68ca5cd.slice/crio-a66cc7b279523aa5ebc0e1e7520abebcd7f2c654cf9364cb7bb37aa42a7b79a4 WatchSource:0}: Error finding container a66cc7b279523aa5ebc0e1e7520abebcd7f2c654cf9364cb7bb37aa42a7b79a4: Status 404 returned error can't find the container with id a66cc7b279523aa5ebc0e1e7520abebcd7f2c654cf9364cb7bb37aa42a7b79a4 Nov 03 15:13:24.822154 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:24.822071 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8rngp" event={"ID":"0e95fe44-e76f-4306-87a2-0ff1e68ca5cd","Type":"ContainerStarted","Data":"a66cc7b279523aa5ebc0e1e7520abebcd7f2c654cf9364cb7bb37aa42a7b79a4"} Nov 03 15:13:26.760565 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:26.760535 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-tgdcp" Nov 03 15:13:27.842902 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:27.842865 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8rngp" event={"ID":"0e95fe44-e76f-4306-87a2-0ff1e68ca5cd","Type":"ContainerStarted","Data":"c27e84dc36165115d3431c2d8e24ac5389ab895fa3bc911dc8c207e49552b347"} Nov 03 15:13:27.978511 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:27.978452 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-8rngp" podStartSLOduration=2.443743831 podStartE2EDuration="4.978436731s" totalImagesPullingTime="2.5346929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:27.975209475 +0000 UTC m=+315.406167281" watchObservedRunningTime="2026-11-03 15:13:27.978436731 +0000 UTC m=+315.409394573" Nov 03 15:13:38.701472 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:13:38.701433 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:13:38.894572 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:38.894538 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg"] Nov 03 15:13:38.899345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:38.899319 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" Nov 03 15:13:38.954206 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:38.954132 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7ddcq\"" Nov 03 15:13:39.004216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.004177 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 15:13:39.004216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.004184 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 15:13:39.008357 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.008325 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/720399f3-c3f4-49fb-8753-ba10d0a9ccea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-s6rdg\" (UID: \"720399f3-c3f4-49fb-8753-ba10d0a9ccea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" Nov 03 15:13:39.008507 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.008457 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxl78\" (UniqueName: \"kubernetes.io/projected/720399f3-c3f4-49fb-8753-ba10d0a9ccea-kube-api-access-wxl78\") pod \"kuadrant-operator-controller-manager-cf8469d7b-s6rdg\" (UID: \"720399f3-c3f4-49fb-8753-ba10d0a9ccea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" Nov 03 15:13:39.009012 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.008992 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg"] Nov 03 15:13:39.057052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.057018 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-22dq6"] Nov 03 15:13:39.060599 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.060577 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-22dq6" Nov 03 15:13:39.103671 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.103630 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-22dq6"] Nov 03 15:13:39.109933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.109908 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/720399f3-c3f4-49fb-8753-ba10d0a9ccea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-s6rdg\" (UID: \"720399f3-c3f4-49fb-8753-ba10d0a9ccea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" Nov 03 15:13:39.110066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.109967 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42jhp\" (UniqueName: \"kubernetes.io/projected/c21126c1-8f77-467b-ba92-fc5010a1c116-kube-api-access-42jhp\") pod \"cert-manager-597b95cb84-22dq6\" (UID: \"c21126c1-8f77-467b-ba92-fc5010a1c116\") " pod="cert-manager/cert-manager-597b95cb84-22dq6" Nov 03 15:13:39.110066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.109996 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c21126c1-8f77-467b-ba92-fc5010a1c116-bound-sa-token\") pod \"cert-manager-597b95cb84-22dq6\" (UID: \"c21126c1-8f77-467b-ba92-fc5010a1c116\") " pod="cert-manager/cert-manager-597b95cb84-22dq6" Nov 03 15:13:39.110066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.110018 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxl78\" (UniqueName: \"kubernetes.io/projected/720399f3-c3f4-49fb-8753-ba10d0a9ccea-kube-api-access-wxl78\") pod \"kuadrant-operator-controller-manager-cf8469d7b-s6rdg\" (UID: \"720399f3-c3f4-49fb-8753-ba10d0a9ccea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" Nov 03 15:13:39.110355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.110333 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/720399f3-c3f4-49fb-8753-ba10d0a9ccea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-s6rdg\" (UID: \"720399f3-c3f4-49fb-8753-ba10d0a9ccea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" Nov 03 15:13:39.113629 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.113603 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jwhks\"" Nov 03 15:13:39.113832 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.113816 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 15:13:39.113917 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.113829 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 15:13:39.174353 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.174313 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxl78\" (UniqueName: \"kubernetes.io/projected/720399f3-c3f4-49fb-8753-ba10d0a9ccea-kube-api-access-wxl78\") pod \"kuadrant-operator-controller-manager-cf8469d7b-s6rdg\" (UID: \"720399f3-c3f4-49fb-8753-ba10d0a9ccea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" Nov 03 15:13:39.211676 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.211595 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42jhp\" (UniqueName: \"kubernetes.io/projected/c21126c1-8f77-467b-ba92-fc5010a1c116-kube-api-access-42jhp\") pod \"cert-manager-597b95cb84-22dq6\" (UID: \"c21126c1-8f77-467b-ba92-fc5010a1c116\") " pod="cert-manager/cert-manager-597b95cb84-22dq6" Nov 03 15:13:39.211676 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.211647 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c21126c1-8f77-467b-ba92-fc5010a1c116-bound-sa-token\") pod \"cert-manager-597b95cb84-22dq6\" (UID: \"c21126c1-8f77-467b-ba92-fc5010a1c116\") " pod="cert-manager/cert-manager-597b95cb84-22dq6" Nov 03 15:13:39.211885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.211683 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" Nov 03 15:13:39.279667 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.279604 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c21126c1-8f77-467b-ba92-fc5010a1c116-bound-sa-token\") pod \"cert-manager-597b95cb84-22dq6\" (UID: \"c21126c1-8f77-467b-ba92-fc5010a1c116\") " pod="cert-manager/cert-manager-597b95cb84-22dq6" Nov 03 15:13:39.281544 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.281514 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42jhp\" (UniqueName: \"kubernetes.io/projected/c21126c1-8f77-467b-ba92-fc5010a1c116-kube-api-access-42jhp\") pod \"cert-manager-597b95cb84-22dq6\" (UID: \"c21126c1-8f77-467b-ba92-fc5010a1c116\") " pod="cert-manager/cert-manager-597b95cb84-22dq6" Nov 03 15:13:39.372789 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.372757 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-22dq6" Nov 03 15:13:39.654208 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.654187 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg"] Nov 03 15:13:39.656435 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:13:39.656408 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720399f3_c3f4_49fb_8753_ba10d0a9ccea.slice/crio-b0282ea20f04682e28f3b82ce4b0b32f2803efb98e66e12ae1c09a70bcb2c8f0 WatchSource:0}: Error finding container b0282ea20f04682e28f3b82ce4b0b32f2803efb98e66e12ae1c09a70bcb2c8f0: Status 404 returned error can't find the container with id b0282ea20f04682e28f3b82ce4b0b32f2803efb98e66e12ae1c09a70bcb2c8f0 Nov 03 15:13:39.761841 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.761812 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-22dq6"] Nov 03 15:13:39.762952 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:13:39.762918 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc21126c1_8f77_467b_ba92_fc5010a1c116.slice/crio-1eae4637ffffea3a9355ed1d61dead3357416b205e7af7afed91f808bd0b4faa WatchSource:0}: Error finding container 1eae4637ffffea3a9355ed1d61dead3357416b205e7af7afed91f808bd0b4faa: Status 404 returned error can't find the container with id 1eae4637ffffea3a9355ed1d61dead3357416b205e7af7afed91f808bd0b4faa Nov 03 15:13:39.906295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.906207 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" event={"ID":"720399f3-c3f4-49fb-8753-ba10d0a9ccea","Type":"ContainerStarted","Data":"b0282ea20f04682e28f3b82ce4b0b32f2803efb98e66e12ae1c09a70bcb2c8f0"} Nov 03 15:13:39.907385 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:39.907361 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-22dq6" event={"ID":"c21126c1-8f77-467b-ba92-fc5010a1c116","Type":"ContainerStarted","Data":"1eae4637ffffea3a9355ed1d61dead3357416b205e7af7afed91f808bd0b4faa"} Nov 03 15:13:41.380979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:41.380938 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8tnvm"] Nov 03 15:13:41.384723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:41.384690 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8tnvm" Nov 03 15:13:41.487311 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:41.487285 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bdrvz\"" Nov 03 15:13:41.491138 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:41.491095 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8tnvm"] Nov 03 15:13:41.535015 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:41.534979 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-662x6\" (UniqueName: \"kubernetes.io/projected/36653232-724e-4a78-b661-5a5c6adc541e-kube-api-access-662x6\") pod \"authorino-operator-7b7bb4bc77-8tnvm\" (UID: \"36653232-724e-4a78-b661-5a5c6adc541e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8tnvm" Nov 03 15:13:41.636310 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:41.636226 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-662x6\" (UniqueName: \"kubernetes.io/projected/36653232-724e-4a78-b661-5a5c6adc541e-kube-api-access-662x6\") pod \"authorino-operator-7b7bb4bc77-8tnvm\" (UID: \"36653232-724e-4a78-b661-5a5c6adc541e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8tnvm" Nov 03 15:13:41.701373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:41.701308 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-662x6\" (UniqueName: \"kubernetes.io/projected/36653232-724e-4a78-b661-5a5c6adc541e-kube-api-access-662x6\") pod \"authorino-operator-7b7bb4bc77-8tnvm\" (UID: \"36653232-724e-4a78-b661-5a5c6adc541e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8tnvm" Nov 03 15:13:42.000780 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:42.000626 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8tnvm" Nov 03 15:13:42.415201 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:42.415167 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8tnvm"] Nov 03 15:13:42.420388 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:13:42.420357 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36653232_724e_4a78_b661_5a5c6adc541e.slice/crio-80280c11c0139521301e1dafb16987676fcf62d6ccb839ef5c5a59150cd98480 WatchSource:0}: Error finding container 80280c11c0139521301e1dafb16987676fcf62d6ccb839ef5c5a59150cd98480: Status 404 returned error can't find the container with id 80280c11c0139521301e1dafb16987676fcf62d6ccb839ef5c5a59150cd98480 Nov 03 15:13:42.931657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:42.931623 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8tnvm" event={"ID":"36653232-724e-4a78-b661-5a5c6adc541e","Type":"ContainerStarted","Data":"80280c11c0139521301e1dafb16987676fcf62d6ccb839ef5c5a59150cd98480"} Nov 03 15:13:44.955041 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:44.954993 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-22dq6" event={"ID":"c21126c1-8f77-467b-ba92-fc5010a1c116","Type":"ContainerStarted","Data":"c296a5c5b1a0732ad63c301c61037d137d2a3fa23f10be49e4a03695c7cf1578"} Nov 03 15:13:45.075389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:45.075312 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-22dq6" podStartSLOduration=2.621772479 podStartE2EDuration="7.075298687s" totalImagesPullingTime="4.453526208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:45.074641758 +0000 UTC m=+332.505599567" watchObservedRunningTime="2026-11-03 15:13:45.075298687 +0000 UTC m=+332.506256492" Nov 03 15:13:45.965668 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:45.965629 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" event={"ID":"720399f3-c3f4-49fb-8753-ba10d0a9ccea","Type":"ContainerStarted","Data":"3436b31185b0f1a1ba2db7d98beb15dce8fbf2dc3f6c0af12e5eb803d6a909c6"} Nov 03 15:13:45.966166 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:45.965818 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" Nov 03 15:13:45.967740 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:45.967694 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8tnvm" event={"ID":"36653232-724e-4a78-b661-5a5c6adc541e","Type":"ContainerStarted","Data":"97478cc201b2f5a5ac46c3793888afc00839f322eab92288d9b0be301fe7fec7"} Nov 03 15:13:45.967885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:45.967771 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8tnvm" Nov 03 15:13:46.192024 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:46.191970 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" podStartSLOduration=2.454506227 podStartE2EDuration="8.191953265s" totalImagesPullingTime="5.737447038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:46.140752973 +0000 UTC m=+333.571710780" watchObservedRunningTime="2026-11-03 15:13:46.191953265 +0000 UTC m=+333.622911071" Nov 03 15:13:46.265208 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:46.265091 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8tnvm" podStartSLOduration=2.334172379 podStartE2EDuration="5.265074491s" totalImagesPullingTime="2.930902112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:46.26215707 +0000 UTC m=+333.693114877" watchObservedRunningTime="2026-11-03 15:13:46.265074491 +0000 UTC m=+333.696032297" Nov 03 15:13:49.717387 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.717341 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h"] Nov 03 15:13:49.720714 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.720668 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.774225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.774196 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-vgdbz\"" Nov 03 15:13:49.818381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.818350 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.818381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.818384 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.818668 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.818405 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.818668 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.818440 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.818668 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.818524 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.818668 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.818549 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.818912 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.818773 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.818912 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.818814 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.818986 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.818949 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8s5k\" (UniqueName: \"kubernetes.io/projected/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-kube-api-access-h8s5k\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.826342 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.826317 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h"] Nov 03 15:13:49.919971 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.919930 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.919971 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.919970 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.920223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.919987 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.920223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.920009 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.920223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.920033 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.920223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.920051 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.920223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.920096 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.920223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.920119 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.920223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.920145 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8s5k\" (UniqueName: \"kubernetes.io/projected/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-kube-api-access-h8s5k\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.920570 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.920430 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.920570 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.920505 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.920798 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.920615 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.920798 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.920642 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.920989 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.920954 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.922582 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.922557 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.922916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.922898 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.984931 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.984852 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:49.984931 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:49.984897 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8s5k\" (UniqueName: \"kubernetes.io/projected/f9ee4847-f5a9-4549-ba96-3e7a9b79db99-kube-api-access-h8s5k\") pod \"data-science-gateway-data-science-gateway-class-6644c85865t8j8h\" (UID: \"f9ee4847-f5a9-4549-ba96-3e7a9b79db99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:50.039023 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:50.038989 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:50.479223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:50.479180 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h"] Nov 03 15:13:50.480506 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:13:50.480482 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ee4847_f5a9_4549_ba96_3e7a9b79db99.slice/crio-fb7e943bb621ccdcadbb0bcb9359bf412175eeaac799fdaff2c6bc1ab3bced46 WatchSource:0}: Error finding container fb7e943bb621ccdcadbb0bcb9359bf412175eeaac799fdaff2c6bc1ab3bced46: Status 404 returned error can't find the container with id fb7e943bb621ccdcadbb0bcb9359bf412175eeaac799fdaff2c6bc1ab3bced46 Nov 03 15:13:50.482664 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:50.482628 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:13:50.482775 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:50.482728 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:13:50.482775 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:50.482763 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:13:50.831913 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:50.831839 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ppb85"] Nov 03 15:13:50.835711 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:50.835685 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ppb85" Nov 03 15:13:50.889575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:50.889543 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7rzzt\"" Nov 03 15:13:50.892166 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:50.892139 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ppb85"] Nov 03 15:13:50.929575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:50.929542 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ths89\" (UniqueName: \"kubernetes.io/projected/9109fa99-e60d-4019-8f59-c61b3a5385a4-kube-api-access-ths89\") pod \"authorino-57f7f8659f-ppb85\" (UID: \"9109fa99-e60d-4019-8f59-c61b3a5385a4\") " pod="kuadrant-system/authorino-57f7f8659f-ppb85" Nov 03 15:13:50.997288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:50.997257 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" event={"ID":"f9ee4847-f5a9-4549-ba96-3e7a9b79db99","Type":"ContainerStarted","Data":"bc750ca0e90711f8eb5745c9d568f0dfb5df1e5064110f611aea1ec91cc123f4"} Nov 03 15:13:50.997466 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:50.997305 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" event={"ID":"f9ee4847-f5a9-4549-ba96-3e7a9b79db99","Type":"ContainerStarted","Data":"fb7e943bb621ccdcadbb0bcb9359bf412175eeaac799fdaff2c6bc1ab3bced46"} Nov 03 15:13:51.030967 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:51.030939 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ths89\" (UniqueName: \"kubernetes.io/projected/9109fa99-e60d-4019-8f59-c61b3a5385a4-kube-api-access-ths89\") pod \"authorino-57f7f8659f-ppb85\" (UID: \"9109fa99-e60d-4019-8f59-c61b3a5385a4\") " pod="kuadrant-system/authorino-57f7f8659f-ppb85" Nov 03 15:13:51.039847 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:51.039826 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:51.044799 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:51.044781 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:51.092671 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:51.092604 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ths89\" (UniqueName: \"kubernetes.io/projected/9109fa99-e60d-4019-8f59-c61b3a5385a4-kube-api-access-ths89\") pod \"authorino-57f7f8659f-ppb85\" (UID: \"9109fa99-e60d-4019-8f59-c61b3a5385a4\") " pod="kuadrant-system/authorino-57f7f8659f-ppb85" Nov 03 15:13:51.120437 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:51.120387 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" podStartSLOduration=2.120373425 podStartE2EDuration="2.120373425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:51.118582499 +0000 UTC m=+338.549540505" watchObservedRunningTime="2026-11-03 15:13:51.120373425 +0000 UTC m=+338.551331231" Nov 03 15:13:51.147888 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:51.147865 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ppb85" Nov 03 15:13:51.521256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:51.521227 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ppb85"] Nov 03 15:13:51.522660 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:13:51.522635 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9109fa99_e60d_4019_8f59_c61b3a5385a4.slice/crio-c1c1b799f8ed22d98f9acb9fe6bdb25fa8721e55b68ba0fd95036b6119b1bb82 WatchSource:0}: Error finding container c1c1b799f8ed22d98f9acb9fe6bdb25fa8721e55b68ba0fd95036b6119b1bb82: Status 404 returned error can't find the container with id c1c1b799f8ed22d98f9acb9fe6bdb25fa8721e55b68ba0fd95036b6119b1bb82 Nov 03 15:13:52.005627 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:52.005584 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ppb85" event={"ID":"9109fa99-e60d-4019-8f59-c61b3a5385a4","Type":"ContainerStarted","Data":"c1c1b799f8ed22d98f9acb9fe6bdb25fa8721e55b68ba0fd95036b6119b1bb82"} Nov 03 15:13:52.006125 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:52.005871 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:52.006817 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:52.006797 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865t8j8h" Nov 03 15:13:55.024458 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:55.024423 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ppb85" event={"ID":"9109fa99-e60d-4019-8f59-c61b3a5385a4","Type":"ContainerStarted","Data":"903b5bd88c8dc5bbe5d1eff1ec4fd90c4b980ba3f291ed9bc026089bce19c186"} Nov 03 15:13:55.140843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:55.140795 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-ppb85" podStartSLOduration=2.364408454 podStartE2EDuration="5.140782941s" totalImagesPullingTime="2.776374487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:55.13854028 +0000 UTC m=+342.569498087" watchObservedRunningTime="2026-11-03 15:13:55.140782941 +0000 UTC m=+342.571740747" Nov 03 15:13:56.975431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:56.975404 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" Nov 03 15:13:56.975857 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:56.975597 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8tnvm" Nov 03 15:13:57.648380 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:57.648347 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68"] Nov 03 15:13:57.651980 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:57.651962 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" Nov 03 15:13:57.673335 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:57.673313 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ppb85"] Nov 03 15:13:57.673459 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:57.673439 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-ppb85" podUID="9109fa99-e60d-4019-8f59-c61b3a5385a4" containerName="authorino" containerID="cri-o://903b5bd88c8dc5bbe5d1eff1ec4fd90c4b980ba3f291ed9bc026089bce19c186" gracePeriod=30 Nov 03 15:13:57.754284 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:57.754249 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68"] Nov 03 15:13:57.793924 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:57.793892 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e54c8082-6a4c-453d-a283-227ba8f71285-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4dh68\" (UID: \"e54c8082-6a4c-453d-a283-227ba8f71285\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" Nov 03 15:13:57.794080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:57.794062 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shm8s\" (UniqueName: \"kubernetes.io/projected/e54c8082-6a4c-453d-a283-227ba8f71285-kube-api-access-shm8s\") pod \"kuadrant-operator-controller-manager-678d87766f-4dh68\" (UID: \"e54c8082-6a4c-453d-a283-227ba8f71285\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" Nov 03 15:13:57.895732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:57.895435 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e54c8082-6a4c-453d-a283-227ba8f71285-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4dh68\" (UID: \"e54c8082-6a4c-453d-a283-227ba8f71285\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" Nov 03 15:13:57.895732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:57.895551 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shm8s\" (UniqueName: \"kubernetes.io/projected/e54c8082-6a4c-453d-a283-227ba8f71285-kube-api-access-shm8s\") pod \"kuadrant-operator-controller-manager-678d87766f-4dh68\" (UID: \"e54c8082-6a4c-453d-a283-227ba8f71285\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" Nov 03 15:13:57.896574 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:57.896527 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e54c8082-6a4c-453d-a283-227ba8f71285-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4dh68\" (UID: \"e54c8082-6a4c-453d-a283-227ba8f71285\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" Nov 03 15:13:57.939158 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:57.939136 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ppb85" Nov 03 15:13:57.960126 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:57.960104 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shm8s\" (UniqueName: \"kubernetes.io/projected/e54c8082-6a4c-453d-a283-227ba8f71285-kube-api-access-shm8s\") pod \"kuadrant-operator-controller-manager-678d87766f-4dh68\" (UID: \"e54c8082-6a4c-453d-a283-227ba8f71285\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" Nov 03 15:13:57.964963 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:57.964947 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" Nov 03 15:13:58.044210 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:58.044176 4290 generic.go:356] "Generic (PLEG): container finished" podID="9109fa99-e60d-4019-8f59-c61b3a5385a4" containerID="903b5bd88c8dc5bbe5d1eff1ec4fd90c4b980ba3f291ed9bc026089bce19c186" exitCode=0 Nov 03 15:13:58.044589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:58.044249 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ppb85" Nov 03 15:13:58.044589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:58.044248 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ppb85" event={"ID":"9109fa99-e60d-4019-8f59-c61b3a5385a4","Type":"ContainerDied","Data":"903b5bd88c8dc5bbe5d1eff1ec4fd90c4b980ba3f291ed9bc026089bce19c186"} Nov 03 15:13:58.044589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:58.044300 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ppb85" event={"ID":"9109fa99-e60d-4019-8f59-c61b3a5385a4","Type":"ContainerDied","Data":"c1c1b799f8ed22d98f9acb9fe6bdb25fa8721e55b68ba0fd95036b6119b1bb82"} Nov 03 15:13:58.044589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:58.044321 4290 scope.go:122] "RemoveContainer" containerID="903b5bd88c8dc5bbe5d1eff1ec4fd90c4b980ba3f291ed9bc026089bce19c186" Nov 03 15:13:58.053666 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:58.053649 4290 scope.go:122] "RemoveContainer" containerID="903b5bd88c8dc5bbe5d1eff1ec4fd90c4b980ba3f291ed9bc026089bce19c186" Nov 03 15:13:58.053953 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:13:58.053932 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"903b5bd88c8dc5bbe5d1eff1ec4fd90c4b980ba3f291ed9bc026089bce19c186\": container with ID starting with 903b5bd88c8dc5bbe5d1eff1ec4fd90c4b980ba3f291ed9bc026089bce19c186 not found: ID does not exist" containerID="903b5bd88c8dc5bbe5d1eff1ec4fd90c4b980ba3f291ed9bc026089bce19c186" Nov 03 15:13:58.054027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:58.053967 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"903b5bd88c8dc5bbe5d1eff1ec4fd90c4b980ba3f291ed9bc026089bce19c186"} err="failed to get container status \"903b5bd88c8dc5bbe5d1eff1ec4fd90c4b980ba3f291ed9bc026089bce19c186\": rpc error: code = NotFound desc = could not find container \"903b5bd88c8dc5bbe5d1eff1ec4fd90c4b980ba3f291ed9bc026089bce19c186\": container with ID starting with 903b5bd88c8dc5bbe5d1eff1ec4fd90c4b980ba3f291ed9bc026089bce19c186 not found: ID does not exist" Nov 03 15:13:58.097684 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:58.097653 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9109fa99-e60d-4019-8f59-c61b3a5385a4-kube-api-access-ths89\" (UniqueName: \"kubernetes.io/projected/9109fa99-e60d-4019-8f59-c61b3a5385a4-kube-api-access-ths89\") pod \"9109fa99-e60d-4019-8f59-c61b3a5385a4\" (UID: \"9109fa99-e60d-4019-8f59-c61b3a5385a4\") " Nov 03 15:13:58.099929 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:58.099902 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9109fa99-e60d-4019-8f59-c61b3a5385a4-kube-api-access-ths89" pod "9109fa99-e60d-4019-8f59-c61b3a5385a4" (UID: "9109fa99-e60d-4019-8f59-c61b3a5385a4"). InnerVolumeSpecName "kube-api-access-ths89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:13:58.199404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:58.199328 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ths89\" (UniqueName: \"kubernetes.io/projected/9109fa99-e60d-4019-8f59-c61b3a5385a4-kube-api-access-ths89\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:13:58.396319 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:58.396289 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68"] Nov 03 15:13:58.397880 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:13:58.397852 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54c8082_6a4c_453d_a283_227ba8f71285.slice/crio-57719c853371d91966ec0212f6253cacef7657e38b1f16ca527d2495c830f68f WatchSource:0}: Error finding container 57719c853371d91966ec0212f6253cacef7657e38b1f16ca527d2495c830f68f: Status 404 returned error can't find the container with id 57719c853371d91966ec0212f6253cacef7657e38b1f16ca527d2495c830f68f Nov 03 15:13:58.472627 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:58.472604 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ppb85"] Nov 03 15:13:58.476901 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:58.476875 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ppb85"] Nov 03 15:13:58.676777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:58.676746 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9109fa99-e60d-4019-8f59-c61b3a5385a4" path="/var/lib/kubelet/pods/9109fa99-e60d-4019-8f59-c61b3a5385a4/volumes" Nov 03 15:13:59.053218 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:59.053179 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" event={"ID":"e54c8082-6a4c-453d-a283-227ba8f71285","Type":"ContainerStarted","Data":"fbaa85fe0d27db114bfde98f42df5509f44a9fecb3fa80636aea7d1348c1420a"} Nov 03 15:13:59.053667 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:59.053233 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" event={"ID":"e54c8082-6a4c-453d-a283-227ba8f71285","Type":"ContainerStarted","Data":"57719c853371d91966ec0212f6253cacef7657e38b1f16ca527d2495c830f68f"} Nov 03 15:13:59.053667 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:59.053349 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" Nov 03 15:13:59.181993 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:13:59.181942 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" podStartSLOduration=2.181926997 podStartE2EDuration="2.181926997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:59.179070029 +0000 UTC m=+346.610027838" watchObservedRunningTime="2026-11-03 15:13:59.181926997 +0000 UTC m=+346.612884802" Nov 03 15:14:01.579845 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.579813 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp"] Nov 03 15:14:01.580409 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.580309 4290 cpu_manager.go:414] "containerMap: removing container" podUID="9109fa99-e60d-4019-8f59-c61b3a5385a4" containerName="authorino" Nov 03 15:14:01.580409 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.580328 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9109fa99-e60d-4019-8f59-c61b3a5385a4" containerName="authorino" Nov 03 15:14:01.580499 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.580463 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="9109fa99-e60d-4019-8f59-c61b3a5385a4" containerName="authorino" Nov 03 15:14:01.584229 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.584214 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:01.637335 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.637308 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-npnq9\"" Nov 03 15:14:01.687611 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.687584 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 15:14:01.687857 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.687827 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 15:14:01.731764 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.731733 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f41479d0-68c6-43c3-8a51-bfef6bcfe828-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-24ckp\" (UID: \"f41479d0-68c6-43c3-8a51-bfef6bcfe828\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:01.731933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.731792 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f41479d0-68c6-43c3-8a51-bfef6bcfe828-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-24ckp\" (UID: \"f41479d0-68c6-43c3-8a51-bfef6bcfe828\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:01.731933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.731910 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f41479d0-68c6-43c3-8a51-bfef6bcfe828-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-24ckp\" (UID: \"f41479d0-68c6-43c3-8a51-bfef6bcfe828\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:01.732027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.731967 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx829\" (UniqueName: \"kubernetes.io/projected/f41479d0-68c6-43c3-8a51-bfef6bcfe828-kube-api-access-jx829\") pod \"kserve-module-controller-manager-7566cf5fc8-24ckp\" (UID: \"f41479d0-68c6-43c3-8a51-bfef6bcfe828\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:01.789561 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.789524 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 15:14:01.791932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.791912 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp"] Nov 03 15:14:01.832755 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.832644 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx829\" (UniqueName: \"kubernetes.io/projected/f41479d0-68c6-43c3-8a51-bfef6bcfe828-kube-api-access-jx829\") pod \"kserve-module-controller-manager-7566cf5fc8-24ckp\" (UID: \"f41479d0-68c6-43c3-8a51-bfef6bcfe828\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:01.832924 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.832766 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f41479d0-68c6-43c3-8a51-bfef6bcfe828-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-24ckp\" (UID: \"f41479d0-68c6-43c3-8a51-bfef6bcfe828\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:01.832924 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.832836 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f41479d0-68c6-43c3-8a51-bfef6bcfe828-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-24ckp\" (UID: \"f41479d0-68c6-43c3-8a51-bfef6bcfe828\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:01.832924 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.832885 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f41479d0-68c6-43c3-8a51-bfef6bcfe828-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-24ckp\" (UID: \"f41479d0-68c6-43c3-8a51-bfef6bcfe828\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:01.833347 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.833327 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f41479d0-68c6-43c3-8a51-bfef6bcfe828-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-24ckp\" (UID: \"f41479d0-68c6-43c3-8a51-bfef6bcfe828\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:01.833590 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.833571 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f41479d0-68c6-43c3-8a51-bfef6bcfe828-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-24ckp\" (UID: \"f41479d0-68c6-43c3-8a51-bfef6bcfe828\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:01.834081 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.834051 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f41479d0-68c6-43c3-8a51-bfef6bcfe828-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-24ckp\" (UID: \"f41479d0-68c6-43c3-8a51-bfef6bcfe828\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:01.895156 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.895124 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx829\" (UniqueName: \"kubernetes.io/projected/f41479d0-68c6-43c3-8a51-bfef6bcfe828-kube-api-access-jx829\") pod \"kserve-module-controller-manager-7566cf5fc8-24ckp\" (UID: \"f41479d0-68c6-43c3-8a51-bfef6bcfe828\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:01.897784 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:01.897754 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:02.332274 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:02.332240 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp"] Nov 03 15:14:02.336210 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:14:02.336176 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41479d0_68c6_43c3_8a51_bfef6bcfe828.slice/crio-d9e63f7a23694dafb2c00b033a28f7a601fc5e4ad112a187e2a148695af31685 WatchSource:0}: Error finding container d9e63f7a23694dafb2c00b033a28f7a601fc5e4ad112a187e2a148695af31685: Status 404 returned error can't find the container with id d9e63f7a23694dafb2c00b033a28f7a601fc5e4ad112a187e2a148695af31685 Nov 03 15:14:03.076571 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:03.076537 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" event={"ID":"f41479d0-68c6-43c3-8a51-bfef6bcfe828","Type":"ContainerStarted","Data":"d9e63f7a23694dafb2c00b033a28f7a601fc5e4ad112a187e2a148695af31685"} Nov 03 15:14:06.096364 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:06.096321 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" event={"ID":"f41479d0-68c6-43c3-8a51-bfef6bcfe828","Type":"ContainerStarted","Data":"75bce0920cd4e56d4eef0931d489a02f979896ac1ad51c946ba57f81780d20bd"} Nov 03 15:14:06.096828 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:06.096478 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:06.268905 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:06.268854 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" podStartSLOduration=2.366289028 podStartE2EDuration="5.268841146s" totalImagesPullingTime="2.902552118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:14:06.266690285 +0000 UTC m=+353.697648091" watchObservedRunningTime="2026-11-03 15:14:06.268841146 +0000 UTC m=+353.699798952" Nov 03 15:14:08.701307 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:14:08.701222 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:14:10.061416 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:10.061384 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" Nov 03 15:14:10.287371 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:10.287334 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg"] Nov 03 15:14:10.287665 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:10.287615 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" podUID="720399f3-c3f4-49fb-8753-ba10d0a9ccea" containerName="manager" containerID="cri-o://3436b31185b0f1a1ba2db7d98beb15dce8fbf2dc3f6c0af12e5eb803d6a909c6" gracePeriod=10 Nov 03 15:14:10.542267 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:10.542245 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" Nov 03 15:14:10.614588 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:10.614561 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/720399f3-c3f4-49fb-8753-ba10d0a9ccea-kube-api-access-wxl78\" (UniqueName: \"kubernetes.io/projected/720399f3-c3f4-49fb-8753-ba10d0a9ccea-kube-api-access-wxl78\") pod \"720399f3-c3f4-49fb-8753-ba10d0a9ccea\" (UID: \"720399f3-c3f4-49fb-8753-ba10d0a9ccea\") " Nov 03 15:14:10.614793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:10.614620 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720399f3-c3f4-49fb-8753-ba10d0a9ccea-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/720399f3-c3f4-49fb-8753-ba10d0a9ccea-extensions-socket-volume\") pod \"720399f3-c3f4-49fb-8753-ba10d0a9ccea\" (UID: \"720399f3-c3f4-49fb-8753-ba10d0a9ccea\") " Nov 03 15:14:10.615095 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:10.615068 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720399f3-c3f4-49fb-8753-ba10d0a9ccea-extensions-socket-volume" pod "720399f3-c3f4-49fb-8753-ba10d0a9ccea" (UID: "720399f3-c3f4-49fb-8753-ba10d0a9ccea"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:10.616642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:10.616618 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720399f3-c3f4-49fb-8753-ba10d0a9ccea-kube-api-access-wxl78" pod "720399f3-c3f4-49fb-8753-ba10d0a9ccea" (UID: "720399f3-c3f4-49fb-8753-ba10d0a9ccea"). InnerVolumeSpecName "kube-api-access-wxl78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:14:10.715904 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:10.715869 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxl78\" (UniqueName: \"kubernetes.io/projected/720399f3-c3f4-49fb-8753-ba10d0a9ccea-kube-api-access-wxl78\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:14:10.715904 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:10.715897 4290 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/720399f3-c3f4-49fb-8753-ba10d0a9ccea-extensions-socket-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:14:11.127077 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:11.127035 4290 generic.go:356] "Generic (PLEG): container finished" podID="720399f3-c3f4-49fb-8753-ba10d0a9ccea" containerID="3436b31185b0f1a1ba2db7d98beb15dce8fbf2dc3f6c0af12e5eb803d6a909c6" exitCode=0 Nov 03 15:14:11.127671 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:11.127147 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" Nov 03 15:14:11.127671 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:11.127157 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" event={"ID":"720399f3-c3f4-49fb-8753-ba10d0a9ccea","Type":"ContainerDied","Data":"3436b31185b0f1a1ba2db7d98beb15dce8fbf2dc3f6c0af12e5eb803d6a909c6"} Nov 03 15:14:11.127671 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:11.127209 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg" event={"ID":"720399f3-c3f4-49fb-8753-ba10d0a9ccea","Type":"ContainerDied","Data":"b0282ea20f04682e28f3b82ce4b0b32f2803efb98e66e12ae1c09a70bcb2c8f0"} Nov 03 15:14:11.127671 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:11.127237 4290 scope.go:122] "RemoveContainer" containerID="3436b31185b0f1a1ba2db7d98beb15dce8fbf2dc3f6c0af12e5eb803d6a909c6" Nov 03 15:14:11.138051 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:11.138026 4290 scope.go:122] "RemoveContainer" containerID="3436b31185b0f1a1ba2db7d98beb15dce8fbf2dc3f6c0af12e5eb803d6a909c6" Nov 03 15:14:11.138368 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:14:11.138349 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3436b31185b0f1a1ba2db7d98beb15dce8fbf2dc3f6c0af12e5eb803d6a909c6\": container with ID starting with 3436b31185b0f1a1ba2db7d98beb15dce8fbf2dc3f6c0af12e5eb803d6a909c6 not found: ID does not exist" containerID="3436b31185b0f1a1ba2db7d98beb15dce8fbf2dc3f6c0af12e5eb803d6a909c6" Nov 03 15:14:11.138433 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:11.138378 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3436b31185b0f1a1ba2db7d98beb15dce8fbf2dc3f6c0af12e5eb803d6a909c6"} err="failed to get container status \"3436b31185b0f1a1ba2db7d98beb15dce8fbf2dc3f6c0af12e5eb803d6a909c6\": rpc error: code = NotFound desc = could not find container \"3436b31185b0f1a1ba2db7d98beb15dce8fbf2dc3f6c0af12e5eb803d6a909c6\": container with ID starting with 3436b31185b0f1a1ba2db7d98beb15dce8fbf2dc3f6c0af12e5eb803d6a909c6 not found: ID does not exist" Nov 03 15:14:11.302577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:11.302534 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg"] Nov 03 15:14:11.352328 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:11.352298 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-s6rdg"] Nov 03 15:14:12.676953 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:12.676921 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="720399f3-c3f4-49fb-8753-ba10d0a9ccea" path="/var/lib/kubelet/pods/720399f3-c3f4-49fb-8753-ba10d0a9ccea/volumes" Nov 03 15:14:14.327752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.327719 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b"] Nov 03 15:14:14.328155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.328134 4290 cpu_manager.go:414] "containerMap: removing container" podUID="720399f3-c3f4-49fb-8753-ba10d0a9ccea" containerName="manager" Nov 03 15:14:14.328155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.328144 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="720399f3-c3f4-49fb-8753-ba10d0a9ccea" containerName="manager" Nov 03 15:14:14.328234 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.328218 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="720399f3-c3f4-49fb-8753-ba10d0a9ccea" containerName="manager" Nov 03 15:14:14.332827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.332812 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:14.352480 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.352447 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8c7fd63-7f3d-4131-8538-fb398095d186-cert\") pod \"jobset-controller-manager-776949766-zcv7b\" (UID: \"e8c7fd63-7f3d-4131-8538-fb398095d186\") " pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:14.352613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.352497 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c7fd63-7f3d-4131-8538-fb398095d186-metrics-certs\") pod \"jobset-controller-manager-776949766-zcv7b\" (UID: \"e8c7fd63-7f3d-4131-8538-fb398095d186\") " pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:14.352613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.352575 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9pwk\" (UniqueName: \"kubernetes.io/projected/e8c7fd63-7f3d-4131-8538-fb398095d186-kube-api-access-v9pwk\") pod \"jobset-controller-manager-776949766-zcv7b\" (UID: \"e8c7fd63-7f3d-4131-8538-fb398095d186\") " pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:14.352731 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.352669 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e8c7fd63-7f3d-4131-8538-fb398095d186-manager-config\") pod \"jobset-controller-manager-776949766-zcv7b\" (UID: \"e8c7fd63-7f3d-4131-8538-fb398095d186\") " pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:14.436670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.436631 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 15:14:14.436670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.436655 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 15:14:14.436670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.436670 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-xrct7\"" Nov 03 15:14:14.436962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.436729 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 15:14:14.439453 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.439415 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b"] Nov 03 15:14:14.453255 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.453235 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8c7fd63-7f3d-4131-8538-fb398095d186-cert\") pod \"jobset-controller-manager-776949766-zcv7b\" (UID: \"e8c7fd63-7f3d-4131-8538-fb398095d186\") " pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:14.453331 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.453271 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c7fd63-7f3d-4131-8538-fb398095d186-metrics-certs\") pod \"jobset-controller-manager-776949766-zcv7b\" (UID: \"e8c7fd63-7f3d-4131-8538-fb398095d186\") " pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:14.453331 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.453296 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9pwk\" (UniqueName: \"kubernetes.io/projected/e8c7fd63-7f3d-4131-8538-fb398095d186-kube-api-access-v9pwk\") pod \"jobset-controller-manager-776949766-zcv7b\" (UID: \"e8c7fd63-7f3d-4131-8538-fb398095d186\") " pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:14.453331 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.453325 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e8c7fd63-7f3d-4131-8538-fb398095d186-manager-config\") pod \"jobset-controller-manager-776949766-zcv7b\" (UID: \"e8c7fd63-7f3d-4131-8538-fb398095d186\") " pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:14.454144 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.454121 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e8c7fd63-7f3d-4131-8538-fb398095d186-manager-config\") pod \"jobset-controller-manager-776949766-zcv7b\" (UID: \"e8c7fd63-7f3d-4131-8538-fb398095d186\") " pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:14.455893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.455871 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8c7fd63-7f3d-4131-8538-fb398095d186-cert\") pod \"jobset-controller-manager-776949766-zcv7b\" (UID: \"e8c7fd63-7f3d-4131-8538-fb398095d186\") " pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:14.456000 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.455962 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c7fd63-7f3d-4131-8538-fb398095d186-metrics-certs\") pod \"jobset-controller-manager-776949766-zcv7b\" (UID: \"e8c7fd63-7f3d-4131-8538-fb398095d186\") " pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:14.515264 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.515237 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9pwk\" (UniqueName: \"kubernetes.io/projected/e8c7fd63-7f3d-4131-8538-fb398095d186-kube-api-access-v9pwk\") pod \"jobset-controller-manager-776949766-zcv7b\" (UID: \"e8c7fd63-7f3d-4131-8538-fb398095d186\") " pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:14.645484 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:14.645454 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:15.029924 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:15.029893 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b"] Nov 03 15:14:15.031189 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:14:15.031164 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c7fd63_7f3d_4131_8538_fb398095d186.slice/crio-958041d2d63dabfe76bc6e46b0d635a8b897b01e4c4e60ca8ba343af0903e1b6 WatchSource:0}: Error finding container 958041d2d63dabfe76bc6e46b0d635a8b897b01e4c4e60ca8ba343af0903e1b6: Status 404 returned error can't find the container with id 958041d2d63dabfe76bc6e46b0d635a8b897b01e4c4e60ca8ba343af0903e1b6 Nov 03 15:14:15.157994 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:15.157960 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" event={"ID":"e8c7fd63-7f3d-4131-8538-fb398095d186","Type":"ContainerStarted","Data":"958041d2d63dabfe76bc6e46b0d635a8b897b01e4c4e60ca8ba343af0903e1b6"} Nov 03 15:14:20.189234 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:20.189192 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" event={"ID":"e8c7fd63-7f3d-4131-8538-fb398095d186","Type":"ContainerStarted","Data":"813d3ebd41a7e480d226b2525e098ffe5dd91f7853dd819d89f2c71220244117"} Nov 03 15:14:20.189678 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:20.189283 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:20.309552 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:20.309498 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" podStartSLOduration=1.7613085640000001 podStartE2EDuration="6.309484391s" totalImagesPullingTime="4.548175827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:14:20.306459058 +0000 UTC m=+367.737416863" watchObservedRunningTime="2026-11-03 15:14:20.309484391 +0000 UTC m=+367.740442195" Nov 03 15:14:27.102666 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:27.102637 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-24ckp" Nov 03 15:14:31.200833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:14:31.200800 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-776949766-zcv7b" Nov 03 15:14:38.700951 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:14:38.700911 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:15:08.706070 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:15:08.705983 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:15:12.356225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.356191 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-8g8dn"] Nov 03 15:15:12.360613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.360591 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:12.413878 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.413855 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 15:15:12.463852 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.463824 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zpmnn\"" Nov 03 15:15:12.463852 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.463842 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 15:15:12.464031 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.463829 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 15:15:12.484476 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.484451 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-metrics-certs\") pod \"maas-controller-579cf7c6d-8g8dn\" (UID: \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\") " pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:12.484640 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.484508 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmd4h\" (UniqueName: \"kubernetes.io/projected/71d11850-4465-438a-a0a6-1dc9a0a0dc85-kube-api-access-wmd4h\") pod \"maas-controller-579cf7c6d-8g8dn\" (UID: \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\") " pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:12.484640 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.484610 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-webhook-cert\") pod \"maas-controller-579cf7c6d-8g8dn\" (UID: \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\") " pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:12.585920 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.585887 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-metrics-certs\") pod \"maas-controller-579cf7c6d-8g8dn\" (UID: \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\") " pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:12.586071 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.585950 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmd4h\" (UniqueName: \"kubernetes.io/projected/71d11850-4465-438a-a0a6-1dc9a0a0dc85-kube-api-access-wmd4h\") pod \"maas-controller-579cf7c6d-8g8dn\" (UID: \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\") " pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:12.586071 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.585990 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-webhook-cert\") pod \"maas-controller-579cf7c6d-8g8dn\" (UID: \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\") " pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:12.586523 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:15:12.586505 4290 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Nov 03 15:15:12.586604 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:15:12.586581 4290 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-webhook-cert podName:71d11850-4465-438a-a0a6-1dc9a0a0dc85 nodeName:}" failed. No retries permitted until 2026-11-03 15:15:13.086558789 +0000 UTC m=+420.517516594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-webhook-cert") pod "maas-controller-579cf7c6d-8g8dn" (UID: "71d11850-4465-438a-a0a6-1dc9a0a0dc85") : secret "maas-controller-webhook-cert" not found Nov 03 15:15:12.588739 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.588718 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-metrics-certs\") pod \"maas-controller-579cf7c6d-8g8dn\" (UID: \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\") " pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:12.650004 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.649974 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmd4h\" (UniqueName: \"kubernetes.io/projected/71d11850-4465-438a-a0a6-1dc9a0a0dc85-kube-api-access-wmd4h\") pod \"maas-controller-579cf7c6d-8g8dn\" (UID: \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\") " pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:12.821453 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.821418 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65f6786cb5-xfb8k"] Nov 03 15:15:12.824054 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.824038 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f6786cb5-xfb8k" Nov 03 15:15:12.888505 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.888476 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngnx9\" (UniqueName: \"kubernetes.io/projected/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-kube-api-access-ngnx9\") pod \"maas-controller-65f6786cb5-xfb8k\" (UID: \"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23\") " pod="opendatahub/maas-controller-65f6786cb5-xfb8k" Nov 03 15:15:12.888663 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.888528 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-webhook-cert\") pod \"maas-controller-65f6786cb5-xfb8k\" (UID: \"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23\") " pod="opendatahub/maas-controller-65f6786cb5-xfb8k" Nov 03 15:15:12.888744 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.888658 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-metrics-certs\") pod \"maas-controller-65f6786cb5-xfb8k\" (UID: \"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23\") " pod="opendatahub/maas-controller-65f6786cb5-xfb8k" Nov 03 15:15:12.989804 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.989720 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-metrics-certs\") pod \"maas-controller-65f6786cb5-xfb8k\" (UID: \"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23\") " pod="opendatahub/maas-controller-65f6786cb5-xfb8k" Nov 03 15:15:12.989804 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.989813 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnx9\" (UniqueName: \"kubernetes.io/projected/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-kube-api-access-ngnx9\") pod \"maas-controller-65f6786cb5-xfb8k\" (UID: \"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23\") " pod="opendatahub/maas-controller-65f6786cb5-xfb8k" Nov 03 15:15:12.990030 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.989852 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-webhook-cert\") pod \"maas-controller-65f6786cb5-xfb8k\" (UID: \"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23\") " pod="opendatahub/maas-controller-65f6786cb5-xfb8k" Nov 03 15:15:12.992353 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.992329 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-metrics-certs\") pod \"maas-controller-65f6786cb5-xfb8k\" (UID: \"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23\") " pod="opendatahub/maas-controller-65f6786cb5-xfb8k" Nov 03 15:15:12.992456 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:12.992337 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-webhook-cert\") pod \"maas-controller-65f6786cb5-xfb8k\" (UID: \"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23\") " pod="opendatahub/maas-controller-65f6786cb5-xfb8k" Nov 03 15:15:13.066919 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:13.066889 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngnx9\" (UniqueName: \"kubernetes.io/projected/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-kube-api-access-ngnx9\") pod \"maas-controller-65f6786cb5-xfb8k\" (UID: \"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23\") " pod="opendatahub/maas-controller-65f6786cb5-xfb8k" Nov 03 15:15:13.091103 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:13.091064 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-webhook-cert\") pod \"maas-controller-579cf7c6d-8g8dn\" (UID: \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\") " pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:13.093388 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:13.093369 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-webhook-cert\") pod \"maas-controller-579cf7c6d-8g8dn\" (UID: \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\") " pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:13.187259 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:13.187231 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zpmnn\"" Nov 03 15:15:13.196288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:13.196270 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f6786cb5-xfb8k" Nov 03 15:15:13.272327 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:13.272250 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:17.180262 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:17.180223 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-8g8dn"] Nov 03 15:15:17.297476 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:17.297442 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f6786cb5-xfb8k"] Nov 03 15:15:17.738994 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:17.738969 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-8g8dn"] Nov 03 15:15:17.741340 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:15:17.741314 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d11850_4465_438a_a0a6_1dc9a0a0dc85.slice/crio-ec29e858a467e31b75be44ecb97995272a041457147d18fe696e3eb490ebb437 WatchSource:0}: Error finding container ec29e858a467e31b75be44ecb97995272a041457147d18fe696e3eb490ebb437: Status 404 returned error can't find the container with id ec29e858a467e31b75be44ecb97995272a041457147d18fe696e3eb490ebb437 Nov 03 15:15:17.985605 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:17.985585 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f6786cb5-xfb8k"] Nov 03 15:15:17.987368 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:15:17.987341 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb8ddc9_3d11_42bb_870f_1d0ca13f8d23.slice/crio-e63ba32c308334b7ba1200bd43c713425a0ed029b25cb26d7a74126be7da61a0 WatchSource:0}: Error finding container e63ba32c308334b7ba1200bd43c713425a0ed029b25cb26d7a74126be7da61a0: Status 404 returned error can't find the container with id e63ba32c308334b7ba1200bd43c713425a0ed029b25cb26d7a74126be7da61a0 Nov 03 15:15:18.513165 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:18.513128 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-8g8dn" event={"ID":"71d11850-4465-438a-a0a6-1dc9a0a0dc85","Type":"ContainerStarted","Data":"ec29e858a467e31b75be44ecb97995272a041457147d18fe696e3eb490ebb437"} Nov 03 15:15:18.514320 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:18.514294 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f6786cb5-xfb8k" event={"ID":"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23","Type":"ContainerStarted","Data":"e63ba32c308334b7ba1200bd43c713425a0ed029b25cb26d7a74126be7da61a0"} Nov 03 15:15:22.061428 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:22.061400 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 15:15:22.546200 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:22.545863 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-8g8dn" event={"ID":"71d11850-4465-438a-a0a6-1dc9a0a0dc85","Type":"ContainerStarted","Data":"616ef944809050e975b965ece36ece32872680df0bd8bcdfb8b9a3dc59928eb4"} Nov 03 15:15:22.546200 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:22.545950 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:22.547539 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:22.547512 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f6786cb5-xfb8k" event={"ID":"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23","Type":"ContainerStarted","Data":"3d1d56bc79b7cf0547e62814ff9316e672d19b8847d1c53e2301fdf6a0d1a01f"} Nov 03 15:15:22.547676 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:22.547564 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65f6786cb5-xfb8k" Nov 03 15:15:22.665464 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:22.665406 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-8g8dn" podStartSLOduration=6.34415549 podStartE2EDuration="10.665392579s" totalImagesPullingTime="4.321237089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:15:22.663632677 +0000 UTC m=+430.094590505" watchObservedRunningTime="2026-11-03 15:15:22.665392579 +0000 UTC m=+430.096350407" Nov 03 15:15:22.798149 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:22.798035 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-65f6786cb5-xfb8k" podStartSLOduration=6.778038532 podStartE2EDuration="10.798016921s" totalImagesPullingTime="4.019978389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:15:22.795412097 +0000 UTC m=+430.226369904" watchObservedRunningTime="2026-11-03 15:15:22.798016921 +0000 UTC m=+430.228974728" Nov 03 15:15:25.983471 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:25.983437 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr"] Nov 03 15:15:25.987135 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:25.987119 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:26.039814 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.039787 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 15:15:26.039946 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.039852 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 15:15:26.089855 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.089820 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 15:15:26.090055 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.089912 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-7dpt7\"" Nov 03 15:15:26.093220 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.093198 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr"] Nov 03 15:15:26.121820 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.121796 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52901e30-cdd3-42e9-9063-20dc5b2f2c66-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hcsvr\" (UID: \"52901e30-cdd3-42e9-9063-20dc5b2f2c66\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:26.121959 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.121931 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7xk\" (UniqueName: \"kubernetes.io/projected/52901e30-cdd3-42e9-9063-20dc5b2f2c66-kube-api-access-qp7xk\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hcsvr\" (UID: \"52901e30-cdd3-42e9-9063-20dc5b2f2c66\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:26.122011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.121958 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52901e30-cdd3-42e9-9063-20dc5b2f2c66-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hcsvr\" (UID: \"52901e30-cdd3-42e9-9063-20dc5b2f2c66\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:26.122049 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.122024 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/52901e30-cdd3-42e9-9063-20dc5b2f2c66-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hcsvr\" (UID: \"52901e30-cdd3-42e9-9063-20dc5b2f2c66\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:26.223194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.223164 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/52901e30-cdd3-42e9-9063-20dc5b2f2c66-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hcsvr\" (UID: \"52901e30-cdd3-42e9-9063-20dc5b2f2c66\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:26.223381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.223209 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52901e30-cdd3-42e9-9063-20dc5b2f2c66-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hcsvr\" (UID: \"52901e30-cdd3-42e9-9063-20dc5b2f2c66\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:26.223381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.223276 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7xk\" (UniqueName: \"kubernetes.io/projected/52901e30-cdd3-42e9-9063-20dc5b2f2c66-kube-api-access-qp7xk\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hcsvr\" (UID: \"52901e30-cdd3-42e9-9063-20dc5b2f2c66\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:26.223381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.223294 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52901e30-cdd3-42e9-9063-20dc5b2f2c66-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hcsvr\" (UID: \"52901e30-cdd3-42e9-9063-20dc5b2f2c66\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:26.223381 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:15:26.223297 4290 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 03 15:15:26.223381 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:15:26.223365 4290 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52901e30-cdd3-42e9-9063-20dc5b2f2c66-proxy-source podName:52901e30-cdd3-42e9-9063-20dc5b2f2c66 nodeName:}" failed. No retries permitted until 2026-11-03 15:15:26.723346356 +0000 UTC m=+434.154304156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/52901e30-cdd3-42e9-9063-20dc5b2f2c66-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" (UID: "52901e30-cdd3-42e9-9063-20dc5b2f2c66") : configmap "usage-tenancy-proxy-source" not found Nov 03 15:15:26.223873 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.223844 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52901e30-cdd3-42e9-9063-20dc5b2f2c66-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hcsvr\" (UID: \"52901e30-cdd3-42e9-9063-20dc5b2f2c66\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:26.225877 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.225859 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52901e30-cdd3-42e9-9063-20dc5b2f2c66-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hcsvr\" (UID: \"52901e30-cdd3-42e9-9063-20dc5b2f2c66\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:26.285667 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.285603 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7xk\" (UniqueName: \"kubernetes.io/projected/52901e30-cdd3-42e9-9063-20dc5b2f2c66-kube-api-access-qp7xk\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hcsvr\" (UID: \"52901e30-cdd3-42e9-9063-20dc5b2f2c66\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:26.727141 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.727106 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/52901e30-cdd3-42e9-9063-20dc5b2f2c66-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hcsvr\" (UID: \"52901e30-cdd3-42e9-9063-20dc5b2f2c66\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:26.728145 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.728119 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/52901e30-cdd3-42e9-9063-20dc5b2f2c66-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hcsvr\" (UID: \"52901e30-cdd3-42e9-9063-20dc5b2f2c66\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:26.900225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:26.900151 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:27.281945 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:27.281920 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr"] Nov 03 15:15:27.283563 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:15:27.283524 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52901e30_cdd3_42e9_9063_20dc5b2f2c66.slice/crio-0649a7c8c46b8e42430c6ce3416845f4eaee32133a90dc7c0dfefc4c68dfa9e3 WatchSource:0}: Error finding container 0649a7c8c46b8e42430c6ce3416845f4eaee32133a90dc7c0dfefc4c68dfa9e3: Status 404 returned error can't find the container with id 0649a7c8c46b8e42430c6ce3416845f4eaee32133a90dc7c0dfefc4c68dfa9e3 Nov 03 15:15:27.577389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:27.577304 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" event={"ID":"52901e30-cdd3-42e9-9063-20dc5b2f2c66","Type":"ContainerStarted","Data":"0649a7c8c46b8e42430c6ce3416845f4eaee32133a90dc7c0dfefc4c68dfa9e3"} Nov 03 15:15:33.556172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:33.556139 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65f6786cb5-xfb8k" Nov 03 15:15:33.556649 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:33.556224 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:33.730056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:33.730023 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-8g8dn"] Nov 03 15:15:33.730272 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:33.730224 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-8g8dn" podUID="71d11850-4465-438a-a0a6-1dc9a0a0dc85" containerName="manager" containerID="cri-o://616ef944809050e975b965ece36ece32872680df0bd8bcdfb8b9a3dc59928eb4" gracePeriod=30 Nov 03 15:15:34.020183 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.020148 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:34.202083 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.202049 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-metrics-certs\") pod \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\" (UID: \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\") " Nov 03 15:15:34.202278 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.202094 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d11850-4465-438a-a0a6-1dc9a0a0dc85-kube-api-access-wmd4h\" (UniqueName: \"kubernetes.io/projected/71d11850-4465-438a-a0a6-1dc9a0a0dc85-kube-api-access-wmd4h\") pod \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\" (UID: \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\") " Nov 03 15:15:34.202557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.202520 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-webhook-cert\" (UniqueName: \"kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-webhook-cert\") pod \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\" (UID: \"71d11850-4465-438a-a0a6-1dc9a0a0dc85\") " Nov 03 15:15:34.205474 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.205443 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-metrics-certs" pod "71d11850-4465-438a-a0a6-1dc9a0a0dc85" (UID: "71d11850-4465-438a-a0a6-1dc9a0a0dc85"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:15:34.206065 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.206014 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d11850-4465-438a-a0a6-1dc9a0a0dc85-kube-api-access-wmd4h" pod "71d11850-4465-438a-a0a6-1dc9a0a0dc85" (UID: "71d11850-4465-438a-a0a6-1dc9a0a0dc85"). InnerVolumeSpecName "kube-api-access-wmd4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:15:34.206752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.206687 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-webhook-cert" pod "71d11850-4465-438a-a0a6-1dc9a0a0dc85" (UID: "71d11850-4465-438a-a0a6-1dc9a0a0dc85"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:15:34.304587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.304540 4290 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-metrics-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:15:34.304587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.304576 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmd4h\" (UniqueName: \"kubernetes.io/projected/71d11850-4465-438a-a0a6-1dc9a0a0dc85-kube-api-access-wmd4h\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:15:34.304587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.304594 4290 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71d11850-4465-438a-a0a6-1dc9a0a0dc85-webhook-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:15:34.625465 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.625423 4290 generic.go:356] "Generic (PLEG): container finished" podID="71d11850-4465-438a-a0a6-1dc9a0a0dc85" containerID="616ef944809050e975b965ece36ece32872680df0bd8bcdfb8b9a3dc59928eb4" exitCode=0 Nov 03 15:15:34.625913 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.625492 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-8g8dn" Nov 03 15:15:34.625913 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.625499 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-8g8dn" event={"ID":"71d11850-4465-438a-a0a6-1dc9a0a0dc85","Type":"ContainerDied","Data":"616ef944809050e975b965ece36ece32872680df0bd8bcdfb8b9a3dc59928eb4"} Nov 03 15:15:34.625913 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.625605 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-8g8dn" event={"ID":"71d11850-4465-438a-a0a6-1dc9a0a0dc85","Type":"ContainerDied","Data":"ec29e858a467e31b75be44ecb97995272a041457147d18fe696e3eb490ebb437"} Nov 03 15:15:34.625913 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.625657 4290 scope.go:122] "RemoveContainer" containerID="616ef944809050e975b965ece36ece32872680df0bd8bcdfb8b9a3dc59928eb4" Nov 03 15:15:34.641171 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.641108 4290 scope.go:122] "RemoveContainer" containerID="616ef944809050e975b965ece36ece32872680df0bd8bcdfb8b9a3dc59928eb4" Nov 03 15:15:34.641531 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:15:34.641477 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616ef944809050e975b965ece36ece32872680df0bd8bcdfb8b9a3dc59928eb4\": container with ID starting with 616ef944809050e975b965ece36ece32872680df0bd8bcdfb8b9a3dc59928eb4 not found: ID does not exist" containerID="616ef944809050e975b965ece36ece32872680df0bd8bcdfb8b9a3dc59928eb4" Nov 03 15:15:34.641531 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.641513 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616ef944809050e975b965ece36ece32872680df0bd8bcdfb8b9a3dc59928eb4"} err="failed to get container status \"616ef944809050e975b965ece36ece32872680df0bd8bcdfb8b9a3dc59928eb4\": rpc error: code = NotFound desc = could not find container \"616ef944809050e975b965ece36ece32872680df0bd8bcdfb8b9a3dc59928eb4\": container with ID starting with 616ef944809050e975b965ece36ece32872680df0bd8bcdfb8b9a3dc59928eb4 not found: ID does not exist" Nov 03 15:15:34.794775 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.794743 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-8g8dn"] Nov 03 15:15:34.798552 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:34.798507 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-8g8dn"] Nov 03 15:15:36.682655 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:36.682615 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d11850-4465-438a-a0a6-1dc9a0a0dc85" path="/var/lib/kubelet/pods/71d11850-4465-438a-a0a6-1dc9a0a0dc85/volumes" Nov 03 15:15:38.704170 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:15:38.704136 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:15:40.668273 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:40.668235 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" event={"ID":"52901e30-cdd3-42e9-9063-20dc5b2f2c66","Type":"ContainerStarted","Data":"26fdd4bff95154e10c4fc4a570be6c1d72d177c3956216b5508da9730cd2177f"} Nov 03 15:15:40.668808 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:40.668306 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:15:40.787662 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:40.787610 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" podStartSLOduration=2.846447156 podStartE2EDuration="15.787595936s" totalImagesPullingTime="12.94114878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:15:40.786677591 +0000 UTC m=+448.217635399" watchObservedRunningTime="2026-11-03 15:15:40.787595936 +0000 UTC m=+448.218553742" Nov 03 15:15:51.679332 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:15:51.679293 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcsvr" Nov 03 15:16:08.703239 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:16:08.703135 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:16:12.160947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.160910 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xjbvr"] Nov 03 15:16:12.161386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.161371 4290 cpu_manager.go:414] "containerMap: removing container" podUID="71d11850-4465-438a-a0a6-1dc9a0a0dc85" containerName="manager" Nov 03 15:16:12.161425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.161387 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d11850-4465-438a-a0a6-1dc9a0a0dc85" containerName="manager" Nov 03 15:16:12.161476 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.161467 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d11850-4465-438a-a0a6-1dc9a0a0dc85" containerName="manager" Nov 03 15:16:12.454674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.454597 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xjbvr"] Nov 03 15:16:12.454843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.454715 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xjbvr" Nov 03 15:16:12.485027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.484994 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px5d5\" (UniqueName: \"kubernetes.io/projected/ec6710b7-93df-44a4-85c2-9d92dc5eb90f-kube-api-access-px5d5\") pod \"authorino-6f6944bcdd-xjbvr\" (UID: \"ec6710b7-93df-44a4-85c2-9d92dc5eb90f\") " pod="kuadrant-system/authorino-6f6944bcdd-xjbvr" Nov 03 15:16:12.507534 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.507509 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7rzzt\"" Nov 03 15:16:12.560041 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.560008 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xjbvr"] Nov 03 15:16:12.560252 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:16:12.560227 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-px5d5], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-xjbvr" podUID="ec6710b7-93df-44a4-85c2-9d92dc5eb90f" Nov 03 15:16:12.585621 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.585594 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px5d5\" (UniqueName: \"kubernetes.io/projected/ec6710b7-93df-44a4-85c2-9d92dc5eb90f-kube-api-access-px5d5\") pod \"authorino-6f6944bcdd-xjbvr\" (UID: \"ec6710b7-93df-44a4-85c2-9d92dc5eb90f\") " pod="kuadrant-system/authorino-6f6944bcdd-xjbvr" Nov 03 15:16:12.647738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.647692 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px5d5\" (UniqueName: \"kubernetes.io/projected/ec6710b7-93df-44a4-85c2-9d92dc5eb90f-kube-api-access-px5d5\") pod \"authorino-6f6944bcdd-xjbvr\" (UID: \"ec6710b7-93df-44a4-85c2-9d92dc5eb90f\") " pod="kuadrant-system/authorino-6f6944bcdd-xjbvr" Nov 03 15:16:12.852938 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.852910 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xjbvr" Nov 03 15:16:12.859320 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.859298 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xjbvr" Nov 03 15:16:12.887427 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.887400 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec6710b7-93df-44a4-85c2-9d92dc5eb90f-kube-api-access-px5d5\" (UniqueName: \"kubernetes.io/projected/ec6710b7-93df-44a4-85c2-9d92dc5eb90f-kube-api-access-px5d5\") pod \"ec6710b7-93df-44a4-85c2-9d92dc5eb90f\" (UID: \"ec6710b7-93df-44a4-85c2-9d92dc5eb90f\") " Nov 03 15:16:12.889864 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.889839 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6710b7-93df-44a4-85c2-9d92dc5eb90f-kube-api-access-px5d5" pod "ec6710b7-93df-44a4-85c2-9d92dc5eb90f" (UID: "ec6710b7-93df-44a4-85c2-9d92dc5eb90f"). InnerVolumeSpecName "kube-api-access-px5d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:16:12.988302 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:12.988233 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px5d5\" (UniqueName: \"kubernetes.io/projected/ec6710b7-93df-44a4-85c2-9d92dc5eb90f-kube-api-access-px5d5\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:16:13.858662 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:13.858623 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xjbvr" Nov 03 15:16:14.098275 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:14.098246 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xjbvr"] Nov 03 15:16:14.100645 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:14.100617 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xjbvr"] Nov 03 15:16:14.676322 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:14.676278 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec6710b7-93df-44a4-85c2-9d92dc5eb90f" path="/var/lib/kubelet/pods/ec6710b7-93df-44a4-85c2-9d92dc5eb90f/volumes" Nov 03 15:16:14.866372 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:14.866315 4290 generic.go:356] "Generic (PLEG): container finished" podID="871baa2b-806e-4033-a553-277c509a3c14" containerID="456abefd97b059f39907e4ac211626e46a1bd334aa9215410394579a01825024" exitCode=0 Nov 03 15:16:14.866808 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:14.866394 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" event={"ID":"871baa2b-806e-4033-a553-277c509a3c14","Type":"ContainerDied","Data":"456abefd97b059f39907e4ac211626e46a1bd334aa9215410394579a01825024"} Nov 03 15:16:14.866862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:14.866822 4290 scope.go:122] "RemoveContainer" containerID="456abefd97b059f39907e4ac211626e46a1bd334aa9215410394579a01825024" Nov 03 15:16:14.868005 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:14.867973 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:16:14.868096 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:14.868051 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:16:14.868096 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:14.868083 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:16:15.874715 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:15.874677 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" event={"ID":"871baa2b-806e-4033-a553-277c509a3c14","Type":"ContainerStarted","Data":"b477955c1879edf80c47ba1678d1cb8f6fa2c0f45862b7f080dcfcda43257503"} Nov 03 15:16:15.956723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:15.954088 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:16:15.956723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:15.954216 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-2sbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 03 15:16:15.956723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:15.954276 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" podUID="871baa2b-806e-4033-a553-277c509a3c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 03 15:16:16.711973 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:16.711942 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dcffb4f64-wg862"] Nov 03 15:16:16.714479 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:16.714460 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dcffb4f64-wg862" Nov 03 15:16:16.770951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:16.770915 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 15:16:16.771514 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:16.771488 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7rzzt\"" Nov 03 15:16:16.774689 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:16.774668 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dcffb4f64-wg862"] Nov 03 15:16:16.821287 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:16.821258 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c4c8e47-9cca-407d-b93f-3fb8aee306e9-tls-cert\") pod \"authorino-5dcffb4f64-wg862\" (UID: \"4c4c8e47-9cca-407d-b93f-3fb8aee306e9\") " pod="kuadrant-system/authorino-5dcffb4f64-wg862" Nov 03 15:16:16.821413 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:16.821329 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49vvq\" (UniqueName: \"kubernetes.io/projected/4c4c8e47-9cca-407d-b93f-3fb8aee306e9-kube-api-access-49vvq\") pod \"authorino-5dcffb4f64-wg862\" (UID: \"4c4c8e47-9cca-407d-b93f-3fb8aee306e9\") " pod="kuadrant-system/authorino-5dcffb4f64-wg862" Nov 03 15:16:16.922167 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:16.922139 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c4c8e47-9cca-407d-b93f-3fb8aee306e9-tls-cert\") pod \"authorino-5dcffb4f64-wg862\" (UID: \"4c4c8e47-9cca-407d-b93f-3fb8aee306e9\") " pod="kuadrant-system/authorino-5dcffb4f64-wg862" Nov 03 15:16:16.922611 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:16.922358 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49vvq\" (UniqueName: \"kubernetes.io/projected/4c4c8e47-9cca-407d-b93f-3fb8aee306e9-kube-api-access-49vvq\") pod \"authorino-5dcffb4f64-wg862\" (UID: \"4c4c8e47-9cca-407d-b93f-3fb8aee306e9\") " pod="kuadrant-system/authorino-5dcffb4f64-wg862" Nov 03 15:16:16.924818 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:16.924798 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c4c8e47-9cca-407d-b93f-3fb8aee306e9-tls-cert\") pod \"authorino-5dcffb4f64-wg862\" (UID: \"4c4c8e47-9cca-407d-b93f-3fb8aee306e9\") " pod="kuadrant-system/authorino-5dcffb4f64-wg862" Nov 03 15:16:16.950734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:16.945014 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-2sbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 03 15:16:16.950734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:16.945063 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" podUID="871baa2b-806e-4033-a553-277c509a3c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 03 15:16:16.984147 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:16.984082 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49vvq\" (UniqueName: \"kubernetes.io/projected/4c4c8e47-9cca-407d-b93f-3fb8aee306e9-kube-api-access-49vvq\") pod \"authorino-5dcffb4f64-wg862\" (UID: \"4c4c8e47-9cca-407d-b93f-3fb8aee306e9\") " pod="kuadrant-system/authorino-5dcffb4f64-wg862" Nov 03 15:16:17.025304 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:17.025276 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dcffb4f64-wg862" Nov 03 15:16:17.459725 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:17.452497 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dcffb4f64-wg862"] Nov 03 15:16:17.465991 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:16:17.465885 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c4c8e47_9cca_407d_b93f_3fb8aee306e9.slice/crio-8ebf9813a69c61e804fea994c9359b658a6173faf5bfb99cc2f0aa756896eb4e WatchSource:0}: Error finding container 8ebf9813a69c61e804fea994c9359b658a6173faf5bfb99cc2f0aa756896eb4e: Status 404 returned error can't find the container with id 8ebf9813a69c61e804fea994c9359b658a6173faf5bfb99cc2f0aa756896eb4e Nov 03 15:16:17.887550 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:17.887514 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dcffb4f64-wg862" event={"ID":"4c4c8e47-9cca-407d-b93f-3fb8aee306e9","Type":"ContainerStarted","Data":"8ebf9813a69c61e804fea994c9359b658a6173faf5bfb99cc2f0aa756896eb4e"} Nov 03 15:16:17.963730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:17.949066 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-2sbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 03 15:16:17.963730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:17.949125 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" podUID="871baa2b-806e-4033-a553-277c509a3c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 03 15:16:18.961727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:18.958673 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-2sbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 03 15:16:18.961727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:18.958746 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" podUID="871baa2b-806e-4033-a553-277c509a3c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 03 15:16:19.904367 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:19.904329 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dcffb4f64-wg862" event={"ID":"4c4c8e47-9cca-407d-b93f-3fb8aee306e9","Type":"ContainerStarted","Data":"86627da390d3659d9ffc20ae5865b03d3fd628220a4222a647f78bad414f35a3"} Nov 03 15:16:19.951100 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:19.951063 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-2sbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 03 15:16:19.951370 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:19.951340 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" podUID="871baa2b-806e-4033-a553-277c509a3c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 03 15:16:20.025585 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:20.025519 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5dcffb4f64-wg862" podStartSLOduration=2.685287425 podStartE2EDuration="4.025504141s" totalImagesPullingTime="1.340216716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:16:20.02282495 +0000 UTC m=+487.453782759" watchObservedRunningTime="2026-11-03 15:16:20.025504141 +0000 UTC m=+487.456461997" Nov 03 15:16:20.945523 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:20.945487 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-2sbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 03 15:16:20.945940 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:20.945545 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" podUID="871baa2b-806e-4033-a553-277c509a3c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 03 15:16:21.949332 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:21.949301 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:16:21.949781 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:21.949612 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:16:21.950393 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:16:21.950375 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" Nov 03 15:16:38.700740 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:16:38.700641 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:17:08.703162 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:17:08.703067 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:17:23.758402 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.758359 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq"] Nov 03 15:17:23.761122 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.761096 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.865246 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.865201 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srbs5\" (UniqueName: \"kubernetes.io/projected/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-kube-api-access-srbs5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.865455 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.865264 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.865455 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.865370 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.865455 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.865415 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.865626 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.865512 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.865626 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.865582 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.867633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.867605 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 15:17:23.867633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.867631 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 15:17:23.917854 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.917821 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-whvzx\"" Nov 03 15:17:23.966650 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.966617 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srbs5\" (UniqueName: \"kubernetes.io/projected/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-kube-api-access-srbs5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.966650 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.966660 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.966941 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.966680 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.966941 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.966844 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.967045 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.966939 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.967045 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.967018 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.967147 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.967121 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.967326 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.967296 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.967456 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.967395 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.967971 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.967938 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 15:17:23.974729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.973945 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq"] Nov 03 15:17:23.978015 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.977992 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:23.980404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:23.980368 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:24.030810 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:24.030739 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srbs5\" (UniqueName: \"kubernetes.io/projected/208146e8-fd8f-4d34-8e96-61c16cd5d4bd-kube-api-access-srbs5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qzlq\" (UID: \"208146e8-fd8f-4d34-8e96-61c16cd5d4bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:24.072950 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:24.072912 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:24.463372 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:17:24.463341 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208146e8_fd8f_4d34_8e96_61c16cd5d4bd.slice/crio-89e6188aa5c182b0e3613e7ec68ddd6ad3f50cc01373d995c243b0836b645b49 WatchSource:0}: Error finding container 89e6188aa5c182b0e3613e7ec68ddd6ad3f50cc01373d995c243b0836b645b49: Status 404 returned error can't find the container with id 89e6188aa5c182b0e3613e7ec68ddd6ad3f50cc01373d995c243b0836b645b49 Nov 03 15:17:24.463714 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:24.463673 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq"] Nov 03 15:17:25.293637 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:25.293598 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" event={"ID":"208146e8-fd8f-4d34-8e96-61c16cd5d4bd","Type":"ContainerStarted","Data":"89e6188aa5c182b0e3613e7ec68ddd6ad3f50cc01373d995c243b0836b645b49"} Nov 03 15:17:26.304988 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:26.304892 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" event={"ID":"208146e8-fd8f-4d34-8e96-61c16cd5d4bd","Type":"ContainerStarted","Data":"bb30f6beae3f6f5ed55ca560de0ff25724ac11c957fb2e0f3cdecf358661b1cb"} Nov 03 15:17:26.474513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:26.474461 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" podStartSLOduration=1.951651929 podStartE2EDuration="3.474447744s" totalImagesPullingTime="1.522795815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:17:26.422401123 +0000 UTC m=+553.853358929" watchObservedRunningTime="2026-11-03 15:17:26.474447744 +0000 UTC m=+553.905405550" Nov 03 15:17:27.074087 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:27.074050 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:34.073680 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:34.073635 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:34.093903 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:34.093867 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:34.363382 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:34.363299 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" Nov 03 15:17:38.702444 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:17:38.702405 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:17:39.250294 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.250258 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td"] Nov 03 15:17:39.252758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.252742 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.355078 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.355049 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 15:17:39.358016 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.357991 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td"] Nov 03 15:17:39.426576 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.426537 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f31f078-0575-4510-bee2-628a20f0caad-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.426825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.426585 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f31f078-0575-4510-bee2-628a20f0caad-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.426825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.426603 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j887v\" (UniqueName: \"kubernetes.io/projected/3f31f078-0575-4510-bee2-628a20f0caad-kube-api-access-j887v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.426825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.426751 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f31f078-0575-4510-bee2-628a20f0caad-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.426825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.426810 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f31f078-0575-4510-bee2-628a20f0caad-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.427057 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.426828 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f31f078-0575-4510-bee2-628a20f0caad-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.527738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.527618 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f31f078-0575-4510-bee2-628a20f0caad-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.527738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.527671 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f31f078-0575-4510-bee2-628a20f0caad-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.527981 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.527751 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f31f078-0575-4510-bee2-628a20f0caad-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.527981 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.527779 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f31f078-0575-4510-bee2-628a20f0caad-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.527981 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.527795 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j887v\" (UniqueName: \"kubernetes.io/projected/3f31f078-0575-4510-bee2-628a20f0caad-kube-api-access-j887v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.527981 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.527811 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f31f078-0575-4510-bee2-628a20f0caad-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.528186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.528113 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f31f078-0575-4510-bee2-628a20f0caad-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.528186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.528148 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f31f078-0575-4510-bee2-628a20f0caad-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.528269 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.528223 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f31f078-0575-4510-bee2-628a20f0caad-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.530233 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.530207 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f31f078-0575-4510-bee2-628a20f0caad-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.530486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.530467 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f31f078-0575-4510-bee2-628a20f0caad-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.589419 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.589389 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j887v\" (UniqueName: \"kubernetes.io/projected/3f31f078-0575-4510-bee2-628a20f0caad-kube-api-access-j887v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td\" (UID: \"3f31f078-0575-4510-bee2-628a20f0caad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:39.864742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:39.864614 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:40.243352 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:40.243326 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td"] Nov 03 15:17:40.244439 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:17:40.244415 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f31f078_0575_4510_bee2_628a20f0caad.slice/crio-3b51694dabebcb0e7de80947a205b807f84905331f60f69245c093ce852d3a3a WatchSource:0}: Error finding container 3b51694dabebcb0e7de80947a205b807f84905331f60f69245c093ce852d3a3a: Status 404 returned error can't find the container with id 3b51694dabebcb0e7de80947a205b807f84905331f60f69245c093ce852d3a3a Nov 03 15:17:40.391742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:40.391682 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" event={"ID":"3f31f078-0575-4510-bee2-628a20f0caad","Type":"ContainerStarted","Data":"3b51694dabebcb0e7de80947a205b807f84905331f60f69245c093ce852d3a3a"} Nov 03 15:17:41.398426 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:41.398384 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" event={"ID":"3f31f078-0575-4510-bee2-628a20f0caad","Type":"ContainerStarted","Data":"2e77dc51ad0b1386bcdb539ba46523d76c70c77924f62112a02251781179665b"} Nov 03 15:17:41.567750 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:41.567666 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" podStartSLOduration=1.965427986 podStartE2EDuration="2.567650197s" totalImagesPullingTime="602.222211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:17:41.566972808 +0000 UTC m=+568.997930614" watchObservedRunningTime="2026-11-03 15:17:41.567650197 +0000 UTC m=+568.998608006" Nov 03 15:17:41.865680 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:41.865595 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:42.269394 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.269361 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qqz7l"] Nov 03 15:17:42.271908 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.271890 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.374180 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.374149 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 15:17:42.377098 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.377075 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qqz7l"] Nov 03 15:17:42.458588 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.458548 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f19618-4cf9-49cf-8416-187d0de0cf0b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.459131 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.458610 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f19618-4cf9-49cf-8416-187d0de0cf0b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.459131 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.458722 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f19618-4cf9-49cf-8416-187d0de0cf0b-home\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.459131 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.458777 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f19618-4cf9-49cf-8416-187d0de0cf0b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.459131 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.458885 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76dtj\" (UniqueName: \"kubernetes.io/projected/72f19618-4cf9-49cf-8416-187d0de0cf0b-kube-api-access-76dtj\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.459131 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.458930 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f19618-4cf9-49cf-8416-187d0de0cf0b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.560300 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.560206 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f19618-4cf9-49cf-8416-187d0de0cf0b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.560300 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.560264 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76dtj\" (UniqueName: \"kubernetes.io/projected/72f19618-4cf9-49cf-8416-187d0de0cf0b-kube-api-access-76dtj\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.560300 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.560297 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f19618-4cf9-49cf-8416-187d0de0cf0b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.560589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.560388 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f19618-4cf9-49cf-8416-187d0de0cf0b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.560589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.560452 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f19618-4cf9-49cf-8416-187d0de0cf0b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.560589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.560484 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f19618-4cf9-49cf-8416-187d0de0cf0b-home\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.560776 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.560664 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f19618-4cf9-49cf-8416-187d0de0cf0b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.560885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.560864 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f19618-4cf9-49cf-8416-187d0de0cf0b-home\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.560969 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.560950 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f19618-4cf9-49cf-8416-187d0de0cf0b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.562934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.562906 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f19618-4cf9-49cf-8416-187d0de0cf0b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.563123 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.563109 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f19618-4cf9-49cf-8416-187d0de0cf0b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.621916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.621882 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76dtj\" (UniqueName: \"kubernetes.io/projected/72f19618-4cf9-49cf-8416-187d0de0cf0b-kube-api-access-76dtj\") pod \"echo-simulator-kserve-64b6f8f96c-qqz7l\" (UID: \"72f19618-4cf9-49cf-8416-187d0de0cf0b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:42.883515 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:42.883472 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:43.275987 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:43.275926 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qqz7l"] Nov 03 15:17:43.277160 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:17:43.277126 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f19618_4cf9_49cf_8416_187d0de0cf0b.slice/crio-4c70fa214da5a47d7265d112d43e094219389b30462f5e78d80c78de8efd2a7e WatchSource:0}: Error finding container 4c70fa214da5a47d7265d112d43e094219389b30462f5e78d80c78de8efd2a7e: Status 404 returned error can't find the container with id 4c70fa214da5a47d7265d112d43e094219389b30462f5e78d80c78de8efd2a7e Nov 03 15:17:43.414273 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:43.414238 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" event={"ID":"72f19618-4cf9-49cf-8416-187d0de0cf0b","Type":"ContainerStarted","Data":"4c70fa214da5a47d7265d112d43e094219389b30462f5e78d80c78de8efd2a7e"} Nov 03 15:17:46.438594 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:46.438560 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" event={"ID":"72f19618-4cf9-49cf-8416-187d0de0cf0b","Type":"ContainerStarted","Data":"2a18fdb574c5afe6af8eb07ed97bf19a8ba462a9bfaf1930f330da55c0916fc3"} Nov 03 15:17:46.606914 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:46.606856 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" podStartSLOduration=1.809389233 podStartE2EDuration="4.606839142s" totalImagesPullingTime="2.797449909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:17:46.555626607 +0000 UTC m=+573.986584413" watchObservedRunningTime="2026-11-03 15:17:46.606839142 +0000 UTC m=+574.037796951" Nov 03 15:17:46.884215 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:46.884185 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:49.865607 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:49.865566 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:49.878626 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:49.878602 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:50.475759 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:50.475725 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td" Nov 03 15:17:52.883796 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:52.883757 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:52.896108 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:52.896082 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:53.515362 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:53.515330 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qqz7l" Nov 03 15:17:54.324517 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.324479 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5"] Nov 03 15:17:54.328401 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.328382 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.389947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.389910 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.390126 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.389961 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.390126 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.390058 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.390126 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.390090 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.390285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.390191 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.390285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.390224 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24535df2-4641-492e-aa5f-9bcdf1273d64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.390285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.390245 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrpxg\" (UniqueName: \"kubernetes.io/projected/24535df2-4641-492e-aa5f-9bcdf1273d64-kube-api-access-nrpxg\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.390386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.390348 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.390386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.390368 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.491160 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.491129 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.491313 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.491174 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24535df2-4641-492e-aa5f-9bcdf1273d64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.491350 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.491303 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrpxg\" (UniqueName: \"kubernetes.io/projected/24535df2-4641-492e-aa5f-9bcdf1273d64-kube-api-access-nrpxg\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.491448 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.491387 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.491448 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.491415 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.491527 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.491458 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.491527 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.491471 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.491608 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.491530 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.491608 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.491587 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.491751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.491613 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.491751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.491731 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.491867 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.491765 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.491867 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.491849 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24535df2-4641-492e-aa5f-9bcdf1273d64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.491981 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.491891 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.494042 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.494014 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.494285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.494264 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.553535 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.553500 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.553690 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.553670 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrpxg\" (UniqueName: \"kubernetes.io/projected/24535df2-4641-492e-aa5f-9bcdf1273d64-kube-api-access-nrpxg\") pod \"maas-default-gateway-openshift-default-596dc9545f-lcdb5\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:54.578012 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.577937 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5"] Nov 03 15:17:54.641175 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:54.641141 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:55.075169 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:55.075137 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5"] Nov 03 15:17:55.077173 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:17:55.077142 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24535df2_4641_492e_aa5f_9bcdf1273d64.slice/crio-9df219d6edc4a8a788339df0974d34089b6a5864755411f38fd919ed43947950 WatchSource:0}: Error finding container 9df219d6edc4a8a788339df0974d34089b6a5864755411f38fd919ed43947950: Status 404 returned error can't find the container with id 9df219d6edc4a8a788339df0974d34089b6a5864755411f38fd919ed43947950 Nov 03 15:17:55.079482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:55.079447 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:17:55.079570 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:55.079520 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:17:55.079570 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:55.079548 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:17:55.525908 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:55.525839 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" event={"ID":"24535df2-4641-492e-aa5f-9bcdf1273d64","Type":"ContainerStarted","Data":"1c72926dda89df4304626630fd477598f9183f2d096bc266174cab3dc676dc71"} Nov 03 15:17:55.526431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:55.525928 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" event={"ID":"24535df2-4641-492e-aa5f-9bcdf1273d64","Type":"ContainerStarted","Data":"9df219d6edc4a8a788339df0974d34089b6a5864755411f38fd919ed43947950"} Nov 03 15:17:55.644726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:55.643811 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:17:55.760726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:55.750867 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" podStartSLOduration=1.750844121 podStartE2EDuration="1.750844121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:17:55.69836705 +0000 UTC m=+583.129324856" watchObservedRunningTime="2026-11-03 15:17:55.750844121 +0000 UTC m=+583.181801933" Nov 03 15:17:56.658728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:56.654823 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lcdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 15:17:56.658728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:56.654889 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" podUID="24535df2-4641-492e-aa5f-9bcdf1273d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 15:17:57.643741 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:57.643121 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lcdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 15:17:57.643741 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:57.643186 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" podUID="24535df2-4641-492e-aa5f-9bcdf1273d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 15:17:58.643133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:58.643094 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lcdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 15:17:58.643768 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:58.643734 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" podUID="24535df2-4641-492e-aa5f-9bcdf1273d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 15:17:59.646113 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:59.646077 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lcdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 15:17:59.646757 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:17:59.646682 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" podUID="24535df2-4641-492e-aa5f-9bcdf1273d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 15:18:00.650738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:00.650169 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lcdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 15:18:00.650738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:00.650236 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" podUID="24535df2-4641-492e-aa5f-9bcdf1273d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 15:18:01.642566 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:01.642526 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lcdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 15:18:01.642790 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:01.642598 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" podUID="24535df2-4641-492e-aa5f-9bcdf1273d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 15:18:03.642748 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:03.642694 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lcdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:18:03.643249 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:03.642809 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" podUID="24535df2-4641-492e-aa5f-9bcdf1273d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:18:04.683894 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:04.683854 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lcdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:18:04.684384 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:04.683923 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" podUID="24535df2-4641-492e-aa5f-9bcdf1273d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:18:05.196394 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:05.196366 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:18:05.196660 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:05.196634 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:18:05.197355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:05.197337 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:18:08.702643 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:18:08.702551 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:18:12.732261 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:12.732224 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:18:12.732676 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:12.732409 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:18:12.734675 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:12.734652 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:18:12.734875 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:12.734861 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:18:20.939115 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:20.939079 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b7867b55-9l2fl"] Nov 03 15:18:20.947425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:20.947400 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b7867b55-9l2fl" Nov 03 15:18:20.992421 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:20.992389 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mbkb\" (UniqueName: \"kubernetes.io/projected/8c0f5828-4470-45ae-af7b-3e2defa2bb3e-kube-api-access-9mbkb\") pod \"authorino-7b7867b55-9l2fl\" (UID: \"8c0f5828-4470-45ae-af7b-3e2defa2bb3e\") " pod="kuadrant-system/authorino-7b7867b55-9l2fl" Nov 03 15:18:20.992574 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:20.992430 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8c0f5828-4470-45ae-af7b-3e2defa2bb3e-tls-cert\") pod \"authorino-7b7867b55-9l2fl\" (UID: \"8c0f5828-4470-45ae-af7b-3e2defa2bb3e\") " pod="kuadrant-system/authorino-7b7867b55-9l2fl" Nov 03 15:18:21.000476 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.000446 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b7867b55-9l2fl"] Nov 03 15:18:21.093215 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.093183 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mbkb\" (UniqueName: \"kubernetes.io/projected/8c0f5828-4470-45ae-af7b-3e2defa2bb3e-kube-api-access-9mbkb\") pod \"authorino-7b7867b55-9l2fl\" (UID: \"8c0f5828-4470-45ae-af7b-3e2defa2bb3e\") " pod="kuadrant-system/authorino-7b7867b55-9l2fl" Nov 03 15:18:21.093398 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.093225 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8c0f5828-4470-45ae-af7b-3e2defa2bb3e-tls-cert\") pod \"authorino-7b7867b55-9l2fl\" (UID: \"8c0f5828-4470-45ae-af7b-3e2defa2bb3e\") " pod="kuadrant-system/authorino-7b7867b55-9l2fl" Nov 03 15:18:21.096120 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.096092 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8c0f5828-4470-45ae-af7b-3e2defa2bb3e-tls-cert\") pod \"authorino-7b7867b55-9l2fl\" (UID: \"8c0f5828-4470-45ae-af7b-3e2defa2bb3e\") " pod="kuadrant-system/authorino-7b7867b55-9l2fl" Nov 03 15:18:21.154874 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.154834 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-kz56c"] Nov 03 15:18:21.158427 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.158400 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mbkb\" (UniqueName: \"kubernetes.io/projected/8c0f5828-4470-45ae-af7b-3e2defa2bb3e-kube-api-access-9mbkb\") pod \"authorino-7b7867b55-9l2fl\" (UID: \"8c0f5828-4470-45ae-af7b-3e2defa2bb3e\") " pod="kuadrant-system/authorino-7b7867b55-9l2fl" Nov 03 15:18:21.159087 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.159065 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" Nov 03 15:18:21.194552 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.194478 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlcr8\" (UniqueName: \"kubernetes.io/projected/372552b1-a937-4067-a251-d284b249970c-kube-api-access-xlcr8\") pod \"ai-gateway-controller-7db955b5ff-kz56c\" (UID: \"372552b1-a937-4067-a251-d284b249970c\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" Nov 03 15:18:21.194552 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.194517 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/372552b1-a937-4067-a251-d284b249970c-metrics-certs\") pod \"ai-gateway-controller-7db955b5ff-kz56c\" (UID: \"372552b1-a937-4067-a251-d284b249970c\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" Nov 03 15:18:21.212913 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.212890 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 15:18:21.213529 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.213103 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 15:18:21.217009 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.216985 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-kz56c"] Nov 03 15:18:21.260751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.260727 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b7867b55-9l2fl" Nov 03 15:18:21.262684 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.262661 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-jwrvw\"" Nov 03 15:18:21.295855 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.295825 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlcr8\" (UniqueName: \"kubernetes.io/projected/372552b1-a937-4067-a251-d284b249970c-kube-api-access-xlcr8\") pod \"ai-gateway-controller-7db955b5ff-kz56c\" (UID: \"372552b1-a937-4067-a251-d284b249970c\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" Nov 03 15:18:21.296010 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.295874 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/372552b1-a937-4067-a251-d284b249970c-metrics-certs\") pod \"ai-gateway-controller-7db955b5ff-kz56c\" (UID: \"372552b1-a937-4067-a251-d284b249970c\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" Nov 03 15:18:21.298652 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.298616 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/372552b1-a937-4067-a251-d284b249970c-metrics-certs\") pod \"ai-gateway-controller-7db955b5ff-kz56c\" (UID: \"372552b1-a937-4067-a251-d284b249970c\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" Nov 03 15:18:21.358282 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.358254 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlcr8\" (UniqueName: \"kubernetes.io/projected/372552b1-a937-4067-a251-d284b249970c-kube-api-access-xlcr8\") pod \"ai-gateway-controller-7db955b5ff-kz56c\" (UID: \"372552b1-a937-4067-a251-d284b249970c\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" Nov 03 15:18:21.481795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.481673 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" Nov 03 15:18:21.646049 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.646023 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b7867b55-9l2fl"] Nov 03 15:18:21.647668 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:18:21.647641 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c0f5828_4470_45ae_af7b_3e2defa2bb3e.slice/crio-c9a532c21c859ebbec12e59b6d677c705a3e982306b1b97b9f91582b3dbee895 WatchSource:0}: Error finding container c9a532c21c859ebbec12e59b6d677c705a3e982306b1b97b9f91582b3dbee895: Status 404 returned error can't find the container with id c9a532c21c859ebbec12e59b6d677c705a3e982306b1b97b9f91582b3dbee895 Nov 03 15:18:21.649035 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.649017 4290 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:18:21.714420 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.714386 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b7867b55-9l2fl" event={"ID":"8c0f5828-4470-45ae-af7b-3e2defa2bb3e","Type":"ContainerStarted","Data":"c9a532c21c859ebbec12e59b6d677c705a3e982306b1b97b9f91582b3dbee895"} Nov 03 15:18:21.861657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:21.861629 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-kz56c"] Nov 03 15:18:21.862580 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:18:21.862557 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372552b1_a937_4067_a251_d284b249970c.slice/crio-7f5193bdd0d67ae62c19edbe6103e915df168aa8484879dce0a73eed25cc2d53 WatchSource:0}: Error finding container 7f5193bdd0d67ae62c19edbe6103e915df168aa8484879dce0a73eed25cc2d53: Status 404 returned error can't find the container with id 7f5193bdd0d67ae62c19edbe6103e915df168aa8484879dce0a73eed25cc2d53 Nov 03 15:18:22.722382 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:22.722346 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" event={"ID":"372552b1-a937-4067-a251-d284b249970c","Type":"ContainerStarted","Data":"7f5193bdd0d67ae62c19edbe6103e915df168aa8484879dce0a73eed25cc2d53"} Nov 03 15:18:27.756825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:27.756781 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" event={"ID":"372552b1-a937-4067-a251-d284b249970c","Type":"ContainerStarted","Data":"b3ecc554097ce2eca8036ecd7dc96edadd9247bc64e1f1dc078021cdaebaf764"} Nov 03 15:18:27.757401 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:27.756869 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" Nov 03 15:18:27.874397 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:27.874344 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" podStartSLOduration=1.5354832680000001 podStartE2EDuration="6.874329711s" totalImagesPullingTime="5.338846443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:18:27.873152993 +0000 UTC m=+615.304110800" watchObservedRunningTime="2026-11-03 15:18:27.874329711 +0000 UTC m=+615.305287516" Nov 03 15:18:28.767264 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:28.767195 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b7867b55-9l2fl" event={"ID":"8c0f5828-4470-45ae-af7b-3e2defa2bb3e","Type":"ContainerStarted","Data":"b6d2b8f8ebf0268fbf68dd4464c753a578fe0a55acb90a1891b052bb8ff26eaa"} Nov 03 15:18:28.884235 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:28.884187 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7b7867b55-9l2fl" podStartSLOduration=1.847091349 podStartE2EDuration="8.884172928s" totalImagesPullingTime="7.037081579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:18:28.882930094 +0000 UTC m=+616.313887901" watchObservedRunningTime="2026-11-03 15:18:28.884172928 +0000 UTC m=+616.315130735" Nov 03 15:18:28.929276 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:28.929233 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dcffb4f64-wg862"] Nov 03 15:18:28.929485 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:28.929457 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5dcffb4f64-wg862" podUID="4c4c8e47-9cca-407d-b93f-3fb8aee306e9" containerName="authorino" containerID="cri-o://86627da390d3659d9ffc20ae5865b03d3fd628220a4222a647f78bad414f35a3" gracePeriod=30 Nov 03 15:18:29.185608 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.185585 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dcffb4f64-wg862" Nov 03 15:18:29.280170 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.280074 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c4c8e47-9cca-407d-b93f-3fb8aee306e9-tls-cert\" (UniqueName: \"kubernetes.io/secret/4c4c8e47-9cca-407d-b93f-3fb8aee306e9-tls-cert\") pod \"4c4c8e47-9cca-407d-b93f-3fb8aee306e9\" (UID: \"4c4c8e47-9cca-407d-b93f-3fb8aee306e9\") " Nov 03 15:18:29.280343 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.280193 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4c8e47-9cca-407d-b93f-3fb8aee306e9-kube-api-access-49vvq\" (UniqueName: \"kubernetes.io/projected/4c4c8e47-9cca-407d-b93f-3fb8aee306e9-kube-api-access-49vvq\") pod \"4c4c8e47-9cca-407d-b93f-3fb8aee306e9\" (UID: \"4c4c8e47-9cca-407d-b93f-3fb8aee306e9\") " Nov 03 15:18:29.283213 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.283177 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4c8e47-9cca-407d-b93f-3fb8aee306e9-kube-api-access-49vvq" pod "4c4c8e47-9cca-407d-b93f-3fb8aee306e9" (UID: "4c4c8e47-9cca-407d-b93f-3fb8aee306e9"). InnerVolumeSpecName "kube-api-access-49vvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:18:29.294247 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.294205 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c4c8e47-9cca-407d-b93f-3fb8aee306e9-tls-cert" pod "4c4c8e47-9cca-407d-b93f-3fb8aee306e9" (UID: "4c4c8e47-9cca-407d-b93f-3fb8aee306e9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:18:29.381443 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.381405 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49vvq\" (UniqueName: \"kubernetes.io/projected/4c4c8e47-9cca-407d-b93f-3fb8aee306e9-kube-api-access-49vvq\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:18:29.381443 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.381439 4290 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c4c8e47-9cca-407d-b93f-3fb8aee306e9-tls-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:18:29.775660 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.775625 4290 generic.go:356] "Generic (PLEG): container finished" podID="4c4c8e47-9cca-407d-b93f-3fb8aee306e9" containerID="86627da390d3659d9ffc20ae5865b03d3fd628220a4222a647f78bad414f35a3" exitCode=0 Nov 03 15:18:29.776108 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.775672 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dcffb4f64-wg862" Nov 03 15:18:29.776108 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.775715 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dcffb4f64-wg862" event={"ID":"4c4c8e47-9cca-407d-b93f-3fb8aee306e9","Type":"ContainerDied","Data":"86627da390d3659d9ffc20ae5865b03d3fd628220a4222a647f78bad414f35a3"} Nov 03 15:18:29.776108 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.775746 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dcffb4f64-wg862" event={"ID":"4c4c8e47-9cca-407d-b93f-3fb8aee306e9","Type":"ContainerDied","Data":"8ebf9813a69c61e804fea994c9359b658a6173faf5bfb99cc2f0aa756896eb4e"} Nov 03 15:18:29.776108 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.775762 4290 scope.go:122] "RemoveContainer" containerID="86627da390d3659d9ffc20ae5865b03d3fd628220a4222a647f78bad414f35a3" Nov 03 15:18:29.786949 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.786925 4290 scope.go:122] "RemoveContainer" containerID="86627da390d3659d9ffc20ae5865b03d3fd628220a4222a647f78bad414f35a3" Nov 03 15:18:29.787227 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:18:29.787206 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86627da390d3659d9ffc20ae5865b03d3fd628220a4222a647f78bad414f35a3\": container with ID starting with 86627da390d3659d9ffc20ae5865b03d3fd628220a4222a647f78bad414f35a3 not found: ID does not exist" containerID="86627da390d3659d9ffc20ae5865b03d3fd628220a4222a647f78bad414f35a3" Nov 03 15:18:29.787358 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.787235 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86627da390d3659d9ffc20ae5865b03d3fd628220a4222a647f78bad414f35a3"} err="failed to get container status \"86627da390d3659d9ffc20ae5865b03d3fd628220a4222a647f78bad414f35a3\": rpc error: code = NotFound desc = could not find container \"86627da390d3659d9ffc20ae5865b03d3fd628220a4222a647f78bad414f35a3\": container with ID starting with 86627da390d3659d9ffc20ae5865b03d3fd628220a4222a647f78bad414f35a3 not found: ID does not exist" Nov 03 15:18:29.941879 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.941842 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dcffb4f64-wg862"] Nov 03 15:18:29.947478 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:29.947410 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dcffb4f64-wg862"] Nov 03 15:18:30.343726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.343612 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vps22"] Nov 03 15:18:30.344425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.344402 4290 cpu_manager.go:414] "containerMap: removing container" podUID="4c4c8e47-9cca-407d-b93f-3fb8aee306e9" containerName="authorino" Nov 03 15:18:30.344477 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.344431 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4c8e47-9cca-407d-b93f-3fb8aee306e9" containerName="authorino" Nov 03 15:18:30.344583 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.344571 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4c8e47-9cca-407d-b93f-3fb8aee306e9" containerName="authorino" Nov 03 15:18:30.349416 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.349392 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vps22" Nov 03 15:18:30.403789 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.403756 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hhwzg\"" Nov 03 15:18:30.407135 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.407109 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vps22"] Nov 03 15:18:30.489806 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.489679 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-f7dkj"] Nov 03 15:18:30.491766 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.491740 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37b293a-544b-4ca5-9810-8bdbc888ca29-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-vps22\" (UID: \"d37b293a-544b-4ca5-9810-8bdbc888ca29\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vps22" Nov 03 15:18:30.491842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.491775 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4shr\" (UniqueName: \"kubernetes.io/projected/d37b293a-544b-4ca5-9810-8bdbc888ca29-kube-api-access-j4shr\") pod \"payload-pre-processing-7b98bb678c-vps22\" (UID: \"d37b293a-544b-4ca5-9810-8bdbc888ca29\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vps22" Nov 03 15:18:30.494183 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.494165 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-f7dkj" Nov 03 15:18:30.498017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.497992 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 15:18:30.551878 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.551840 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-f7dkj"] Nov 03 15:18:30.593426 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.593387 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-f7dkj\" (UID: \"0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-f7dkj" Nov 03 15:18:30.593592 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.593501 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4cph\" (UniqueName: \"kubernetes.io/projected/0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8-kube-api-access-w4cph\") pod \"payload-processing-55fbbbf9dc-f7dkj\" (UID: \"0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-f7dkj" Nov 03 15:18:30.593646 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.593601 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37b293a-544b-4ca5-9810-8bdbc888ca29-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-vps22\" (UID: \"d37b293a-544b-4ca5-9810-8bdbc888ca29\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vps22" Nov 03 15:18:30.593710 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.593674 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4shr\" (UniqueName: \"kubernetes.io/projected/d37b293a-544b-4ca5-9810-8bdbc888ca29-kube-api-access-j4shr\") pod \"payload-pre-processing-7b98bb678c-vps22\" (UID: \"d37b293a-544b-4ca5-9810-8bdbc888ca29\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vps22" Nov 03 15:18:30.594424 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.594377 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37b293a-544b-4ca5-9810-8bdbc888ca29-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-vps22\" (UID: \"d37b293a-544b-4ca5-9810-8bdbc888ca29\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vps22" Nov 03 15:18:30.656089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.656050 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4shr\" (UniqueName: \"kubernetes.io/projected/d37b293a-544b-4ca5-9810-8bdbc888ca29-kube-api-access-j4shr\") pod \"payload-pre-processing-7b98bb678c-vps22\" (UID: \"d37b293a-544b-4ca5-9810-8bdbc888ca29\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vps22" Nov 03 15:18:30.667755 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.667728 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vps22" Nov 03 15:18:30.677885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.677856 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4c8e47-9cca-407d-b93f-3fb8aee306e9" path="/var/lib/kubelet/pods/4c4c8e47-9cca-407d-b93f-3fb8aee306e9/volumes" Nov 03 15:18:30.694946 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.694910 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4cph\" (UniqueName: \"kubernetes.io/projected/0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8-kube-api-access-w4cph\") pod \"payload-processing-55fbbbf9dc-f7dkj\" (UID: \"0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-f7dkj" Nov 03 15:18:30.695133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.695112 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-f7dkj\" (UID: \"0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-f7dkj" Nov 03 15:18:30.695792 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.695772 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-f7dkj\" (UID: \"0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-f7dkj" Nov 03 15:18:30.760723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.760615 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4cph\" (UniqueName: \"kubernetes.io/projected/0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8-kube-api-access-w4cph\") pod \"payload-processing-55fbbbf9dc-f7dkj\" (UID: \"0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-f7dkj" Nov 03 15:18:30.834816 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:30.834777 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-f7dkj" Nov 03 15:18:31.058098 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:31.058070 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vps22"] Nov 03 15:18:31.060415 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:18:31.060390 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37b293a_544b_4ca5_9810_8bdbc888ca29.slice/crio-122ebf7ddb032c496de6f82cb74d9e73779cd05428d62d1de4301096b1376caf WatchSource:0}: Error finding container 122ebf7ddb032c496de6f82cb74d9e73779cd05428d62d1de4301096b1376caf: Status 404 returned error can't find the container with id 122ebf7ddb032c496de6f82cb74d9e73779cd05428d62d1de4301096b1376caf Nov 03 15:18:31.223386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:31.223357 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-f7dkj"] Nov 03 15:18:31.225630 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:18:31.225599 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cae1c25_ed2d_4b5a_b4dc_0f87dfc219e8.slice/crio-1d77c6c2ec1d9015fd855489c760c91d319997d38e4995fd514cb82c3d1e2336 WatchSource:0}: Error finding container 1d77c6c2ec1d9015fd855489c760c91d319997d38e4995fd514cb82c3d1e2336: Status 404 returned error can't find the container with id 1d77c6c2ec1d9015fd855489c760c91d319997d38e4995fd514cb82c3d1e2336 Nov 03 15:18:31.791475 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:31.791432 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-f7dkj" event={"ID":"0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8","Type":"ContainerStarted","Data":"1d77c6c2ec1d9015fd855489c760c91d319997d38e4995fd514cb82c3d1e2336"} Nov 03 15:18:31.792846 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:31.792813 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vps22" event={"ID":"d37b293a-544b-4ca5-9810-8bdbc888ca29","Type":"ContainerStarted","Data":"122ebf7ddb032c496de6f82cb74d9e73779cd05428d62d1de4301096b1376caf"} Nov 03 15:18:34.829921 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:34.829880 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vps22" event={"ID":"d37b293a-544b-4ca5-9810-8bdbc888ca29","Type":"ContainerStarted","Data":"aa65f3f6498986367e07db0cf7a4f96d358c1e2195b720fb4b71e7b2e9ef8800"} Nov 03 15:18:34.830389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:34.829955 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vps22" Nov 03 15:18:34.831736 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:34.831709 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-f7dkj" event={"ID":"0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8","Type":"ContainerStarted","Data":"e1f12bc85d92ab0b448647c0f150b5152980cec58da40b177fb9f96b02015267"} Nov 03 15:18:34.831874 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:34.831838 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-f7dkj" Nov 03 15:18:34.947730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:34.947608 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vps22" podStartSLOduration=1.381771568 podStartE2EDuration="4.947592702s" totalImagesPullingTime="3.565821134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:18:34.945746659 +0000 UTC m=+622.376704468" watchObservedRunningTime="2026-11-03 15:18:34.947592702 +0000 UTC m=+622.378550507" Nov 03 15:18:35.063750 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:35.063678 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-f7dkj" podStartSLOduration=1.6208399089999999 podStartE2EDuration="5.063663846s" totalImagesPullingTime="3.442823937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:18:35.061141114 +0000 UTC m=+622.492098921" watchObservedRunningTime="2026-11-03 15:18:35.063663846 +0000 UTC m=+622.494621652" Nov 03 15:18:38.701638 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:18:38.701594 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:18:38.770172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:38.770143 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" Nov 03 15:18:39.138643 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:39.138607 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-59f8d677c6-hwwtv"] Nov 03 15:18:39.142758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:39.142740 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-59f8d677c6-hwwtv" Nov 03 15:18:39.203371 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:39.203338 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-59f8d677c6-hwwtv"] Nov 03 15:18:39.327567 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:39.327526 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfwwz\" (UniqueName: \"kubernetes.io/projected/060263ec-48f6-48ab-b4aa-04aa578327ec-kube-api-access-lfwwz\") pod \"ai-gateway-controller-59f8d677c6-hwwtv\" (UID: \"060263ec-48f6-48ab-b4aa-04aa578327ec\") " pod="opendatahub/ai-gateway-controller-59f8d677c6-hwwtv" Nov 03 15:18:39.327779 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:39.327605 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060263ec-48f6-48ab-b4aa-04aa578327ec-metrics-certs\") pod \"ai-gateway-controller-59f8d677c6-hwwtv\" (UID: \"060263ec-48f6-48ab-b4aa-04aa578327ec\") " pod="opendatahub/ai-gateway-controller-59f8d677c6-hwwtv" Nov 03 15:18:39.429223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:39.429142 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060263ec-48f6-48ab-b4aa-04aa578327ec-metrics-certs\") pod \"ai-gateway-controller-59f8d677c6-hwwtv\" (UID: \"060263ec-48f6-48ab-b4aa-04aa578327ec\") " pod="opendatahub/ai-gateway-controller-59f8d677c6-hwwtv" Nov 03 15:18:39.429379 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:39.429248 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwwz\" (UniqueName: \"kubernetes.io/projected/060263ec-48f6-48ab-b4aa-04aa578327ec-kube-api-access-lfwwz\") pod \"ai-gateway-controller-59f8d677c6-hwwtv\" (UID: \"060263ec-48f6-48ab-b4aa-04aa578327ec\") " pod="opendatahub/ai-gateway-controller-59f8d677c6-hwwtv" Nov 03 15:18:39.431971 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:39.431946 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060263ec-48f6-48ab-b4aa-04aa578327ec-metrics-certs\") pod \"ai-gateway-controller-59f8d677c6-hwwtv\" (UID: \"060263ec-48f6-48ab-b4aa-04aa578327ec\") " pod="opendatahub/ai-gateway-controller-59f8d677c6-hwwtv" Nov 03 15:18:39.492845 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:39.492819 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfwwz\" (UniqueName: \"kubernetes.io/projected/060263ec-48f6-48ab-b4aa-04aa578327ec-kube-api-access-lfwwz\") pod \"ai-gateway-controller-59f8d677c6-hwwtv\" (UID: \"060263ec-48f6-48ab-b4aa-04aa578327ec\") " pod="opendatahub/ai-gateway-controller-59f8d677c6-hwwtv" Nov 03 15:18:39.755815 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:39.755734 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-59f8d677c6-hwwtv" Nov 03 15:18:40.135908 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:40.135874 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-59f8d677c6-hwwtv"] Nov 03 15:18:40.139346 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:18:40.139319 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060263ec_48f6_48ab_b4aa_04aa578327ec.slice/crio-7add683e9c5eaeef074887a41c93a193ca3d1528694691d24285e33ebd3aa606 WatchSource:0}: Error finding container 7add683e9c5eaeef074887a41c93a193ca3d1528694691d24285e33ebd3aa606: Status 404 returned error can't find the container with id 7add683e9c5eaeef074887a41c93a193ca3d1528694691d24285e33ebd3aa606 Nov 03 15:18:40.869572 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:40.869526 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-59f8d677c6-hwwtv" event={"ID":"060263ec-48f6-48ab-b4aa-04aa578327ec","Type":"ContainerStarted","Data":"7add683e9c5eaeef074887a41c93a193ca3d1528694691d24285e33ebd3aa606"} Nov 03 15:18:42.883091 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:42.883052 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-59f8d677c6-hwwtv" event={"ID":"060263ec-48f6-48ab-b4aa-04aa578327ec","Type":"ContainerStarted","Data":"f144b25941cd3e47606d331055af73a63e9d2e2c502f00f0eaed2e7d91a6f9e5"} Nov 03 15:18:42.883520 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:42.883143 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-59f8d677c6-hwwtv" Nov 03 15:18:43.006016 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:43.005967 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-59f8d677c6-hwwtv" podStartSLOduration=2.183668133 podStartE2EDuration="4.005953155s" totalImagesPullingTime="1.822285022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:18:43.002825087 +0000 UTC m=+630.433782894" watchObservedRunningTime="2026-11-03 15:18:43.005953155 +0000 UTC m=+630.436910961" Nov 03 15:18:45.840081 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:45.840051 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vps22" Nov 03 15:18:45.840464 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:45.840111 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-f7dkj" Nov 03 15:18:53.890884 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:53.890849 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-59f8d677c6-hwwtv" Nov 03 15:18:54.065507 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.065477 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-kz56c"] Nov 03 15:18:54.065743 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.065693 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" podUID="372552b1-a937-4067-a251-d284b249970c" containerName="manager" containerID="cri-o://b3ecc554097ce2eca8036ecd7dc96edadd9247bc64e1f1dc078021cdaebaf764" gracePeriod=30 Nov 03 15:18:54.321212 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.321188 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" Nov 03 15:18:54.373056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.373030 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/372552b1-a937-4067-a251-d284b249970c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/372552b1-a937-4067-a251-d284b249970c-metrics-certs\") pod \"372552b1-a937-4067-a251-d284b249970c\" (UID: \"372552b1-a937-4067-a251-d284b249970c\") " Nov 03 15:18:54.373212 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.373194 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372552b1-a937-4067-a251-d284b249970c-kube-api-access-xlcr8\" (UniqueName: \"kubernetes.io/projected/372552b1-a937-4067-a251-d284b249970c-kube-api-access-xlcr8\") pod \"372552b1-a937-4067-a251-d284b249970c\" (UID: \"372552b1-a937-4067-a251-d284b249970c\") " Nov 03 15:18:54.375232 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.375207 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372552b1-a937-4067-a251-d284b249970c-kube-api-access-xlcr8" pod "372552b1-a937-4067-a251-d284b249970c" (UID: "372552b1-a937-4067-a251-d284b249970c"). InnerVolumeSpecName "kube-api-access-xlcr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:18:54.375494 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.375475 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/372552b1-a937-4067-a251-d284b249970c-metrics-certs" pod "372552b1-a937-4067-a251-d284b249970c" (UID: "372552b1-a937-4067-a251-d284b249970c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:18:54.475569 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.475484 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlcr8\" (UniqueName: \"kubernetes.io/projected/372552b1-a937-4067-a251-d284b249970c-kube-api-access-xlcr8\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:18:54.475569 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.475511 4290 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/372552b1-a937-4067-a251-d284b249970c-metrics-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:18:54.952846 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.952814 4290 generic.go:356] "Generic (PLEG): container finished" podID="372552b1-a937-4067-a251-d284b249970c" containerID="b3ecc554097ce2eca8036ecd7dc96edadd9247bc64e1f1dc078021cdaebaf764" exitCode=0 Nov 03 15:18:54.953320 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.952855 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" Nov 03 15:18:54.953320 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.952895 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" event={"ID":"372552b1-a937-4067-a251-d284b249970c","Type":"ContainerDied","Data":"b3ecc554097ce2eca8036ecd7dc96edadd9247bc64e1f1dc078021cdaebaf764"} Nov 03 15:18:54.953320 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.952958 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7db955b5ff-kz56c" event={"ID":"372552b1-a937-4067-a251-d284b249970c","Type":"ContainerDied","Data":"7f5193bdd0d67ae62c19edbe6103e915df168aa8484879dce0a73eed25cc2d53"} Nov 03 15:18:54.953320 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.952993 4290 scope.go:122] "RemoveContainer" containerID="b3ecc554097ce2eca8036ecd7dc96edadd9247bc64e1f1dc078021cdaebaf764" Nov 03 15:18:54.962541 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.962523 4290 scope.go:122] "RemoveContainer" containerID="b3ecc554097ce2eca8036ecd7dc96edadd9247bc64e1f1dc078021cdaebaf764" Nov 03 15:18:54.962843 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:18:54.962819 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ecc554097ce2eca8036ecd7dc96edadd9247bc64e1f1dc078021cdaebaf764\": container with ID starting with b3ecc554097ce2eca8036ecd7dc96edadd9247bc64e1f1dc078021cdaebaf764 not found: ID does not exist" containerID="b3ecc554097ce2eca8036ecd7dc96edadd9247bc64e1f1dc078021cdaebaf764" Nov 03 15:18:54.962934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:54.962850 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ecc554097ce2eca8036ecd7dc96edadd9247bc64e1f1dc078021cdaebaf764"} err="failed to get container status \"b3ecc554097ce2eca8036ecd7dc96edadd9247bc64e1f1dc078021cdaebaf764\": rpc error: code = NotFound desc = could not find container \"b3ecc554097ce2eca8036ecd7dc96edadd9247bc64e1f1dc078021cdaebaf764\": container with ID starting with b3ecc554097ce2eca8036ecd7dc96edadd9247bc64e1f1dc078021cdaebaf764 not found: ID does not exist" Nov 03 15:18:55.126092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:55.126067 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-kz56c"] Nov 03 15:18:55.177082 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:55.177051 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-kz56c"] Nov 03 15:18:56.677546 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:18:56.677509 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="372552b1-a937-4067-a251-d284b249970c" path="/var/lib/kubelet/pods/372552b1-a937-4067-a251-d284b249970c/volumes" Nov 03 15:19:08.703228 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:19:08.703146 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:19:34.226645 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.226560 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm"] Nov 03 15:19:34.227177 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.227159 4290 cpu_manager.go:414] "containerMap: removing container" podUID="372552b1-a937-4067-a251-d284b249970c" containerName="manager" Nov 03 15:19:34.227226 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.227178 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="372552b1-a937-4067-a251-d284b249970c" containerName="manager" Nov 03 15:19:34.227261 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.227254 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="372552b1-a937-4067-a251-d284b249970c" containerName="manager" Nov 03 15:19:34.230581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.230563 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" Nov 03 15:19:34.290551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.290506 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm"] Nov 03 15:19:34.337887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.337858 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-34e8c5\"" Nov 03 15:19:34.348038 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.348017 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-34e8c5-dockercfg-7r8xm\"" Nov 03 15:19:34.366323 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.366291 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d27nh\" (UniqueName: \"kubernetes.io/projected/247dd294-ca8a-49ca-a75e-a2d65ceb7991-kube-api-access-d27nh\") pod \"payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm\" (UID: \"247dd294-ca8a-49ca-a75e-a2d65ceb7991\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" Nov 03 15:19:34.366461 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.366410 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247dd294-ca8a-49ca-a75e-a2d65ceb7991-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm\" (UID: \"247dd294-ca8a-49ca-a75e-a2d65ceb7991\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" Nov 03 15:19:34.374089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.374056 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4"] Nov 03 15:19:34.377848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.377834 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" Nov 03 15:19:34.438111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.438079 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4"] Nov 03 15:19:34.467528 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.467498 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d27nh\" (UniqueName: \"kubernetes.io/projected/247dd294-ca8a-49ca-a75e-a2d65ceb7991-kube-api-access-d27nh\") pod \"payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm\" (UID: \"247dd294-ca8a-49ca-a75e-a2d65ceb7991\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" Nov 03 15:19:34.467693 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.467596 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247dd294-ca8a-49ca-a75e-a2d65ceb7991-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm\" (UID: \"247dd294-ca8a-49ca-a75e-a2d65ceb7991\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" Nov 03 15:19:34.468306 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.468284 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247dd294-ca8a-49ca-a75e-a2d65ceb7991-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm\" (UID: \"247dd294-ca8a-49ca-a75e-a2d65ceb7991\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" Nov 03 15:19:34.530721 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.530641 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d27nh\" (UniqueName: \"kubernetes.io/projected/247dd294-ca8a-49ca-a75e-a2d65ceb7991-kube-api-access-d27nh\") pod \"payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm\" (UID: \"247dd294-ca8a-49ca-a75e-a2d65ceb7991\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" Nov 03 15:19:34.543468 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.543443 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" Nov 03 15:19:34.568862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.568803 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4\" (UID: \"b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" Nov 03 15:19:34.569007 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.568986 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb4fb\" (UniqueName: \"kubernetes.io/projected/b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8-kube-api-access-jb4fb\") pod \"payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4\" (UID: \"b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" Nov 03 15:19:34.669523 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.669493 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb4fb\" (UniqueName: \"kubernetes.io/projected/b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8-kube-api-access-jb4fb\") pod \"payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4\" (UID: \"b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" Nov 03 15:19:34.669665 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.669544 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4\" (UID: \"b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" Nov 03 15:19:34.670207 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.670188 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4\" (UID: \"b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" Nov 03 15:19:34.732680 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.732647 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb4fb\" (UniqueName: \"kubernetes.io/projected/b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8-kube-api-access-jb4fb\") pod \"payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4\" (UID: \"b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" Nov 03 15:19:34.926411 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.926379 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm"] Nov 03 15:19:34.933011 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:19:34.932979 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247dd294_ca8a_49ca_a75e_a2d65ceb7991.slice/crio-c95121354e0fd513132631f2f5cb578bf047574c6c21144c3d8674bd52e4e3ae WatchSource:0}: Error finding container c95121354e0fd513132631f2f5cb578bf047574c6c21144c3d8674bd52e4e3ae: Status 404 returned error can't find the container with id c95121354e0fd513132631f2f5cb578bf047574c6c21144c3d8674bd52e4e3ae Nov 03 15:19:34.991481 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:34.991461 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" Nov 03 15:19:35.209920 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:35.209824 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" event={"ID":"247dd294-ca8a-49ca-a75e-a2d65ceb7991","Type":"ContainerStarted","Data":"6938df5dbfcdf3a0eebe0897ab060eabcb2e09fac158eb4ce8c522ad4594e8b4"} Nov 03 15:19:35.209920 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:35.209879 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" event={"ID":"247dd294-ca8a-49ca-a75e-a2d65ceb7991","Type":"ContainerStarted","Data":"c95121354e0fd513132631f2f5cb578bf047574c6c21144c3d8674bd52e4e3ae"} Nov 03 15:19:35.209920 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:35.209904 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" Nov 03 15:19:35.331641 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:35.331588 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" podStartSLOduration=1.331574474 podStartE2EDuration="1.331574474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:35.329408873 +0000 UTC m=+682.760366679" watchObservedRunningTime="2026-11-03 15:19:35.331574474 +0000 UTC m=+682.762532277" Nov 03 15:19:35.370150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:35.370129 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4"] Nov 03 15:19:35.372576 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:19:35.372553 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12fd72d_c28d_4fe3_b0d4_c2f97ca1c5c8.slice/crio-4471114e042da66ee3919714c4e97014c1d6e4e33b37b655316e0b13c5d81d43 WatchSource:0}: Error finding container 4471114e042da66ee3919714c4e97014c1d6e4e33b37b655316e0b13c5d81d43: Status 404 returned error can't find the container with id 4471114e042da66ee3919714c4e97014c1d6e4e33b37b655316e0b13c5d81d43 Nov 03 15:19:36.223773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:36.223691 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" event={"ID":"b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8","Type":"ContainerStarted","Data":"a1aa01937825caba9d90bdf28fdcbe8ba4f808e90ba344e077e6d7f4378a5f3d"} Nov 03 15:19:36.223984 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:36.223888 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" event={"ID":"b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8","Type":"ContainerStarted","Data":"4471114e042da66ee3919714c4e97014c1d6e4e33b37b655316e0b13c5d81d43"} Nov 03 15:19:36.224145 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:36.224128 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" Nov 03 15:19:36.345874 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:36.345816 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" podStartSLOduration=2.345800615 podStartE2EDuration="2.345800615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:36.343550099 +0000 UTC m=+683.774507904" watchObservedRunningTime="2026-11-03 15:19:36.345800615 +0000 UTC m=+683.776758420" Nov 03 15:19:36.909124 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:36.909087 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9"] Nov 03 15:19:36.913240 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:36.913218 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:36.991565 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:36.991536 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:36.991777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:36.991573 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:36.991777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:36.991666 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2nmj\" (UniqueName: \"kubernetes.io/projected/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-kube-api-access-w2nmj\") pod \"maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:36.991921 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:36.991802 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-metrics-certs\") pod \"maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:37.019789 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.019764 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-9855ff-serving-cert\"" Nov 03 15:19:37.019931 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.019828 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p84nb\"" Nov 03 15:19:37.019931 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.019850 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:19:37.020017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.019871 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 15:19:37.020017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.019897 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:19:37.022990 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.022972 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9"] Nov 03 15:19:37.092372 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.092341 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-metrics-certs\") pod \"maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:37.092550 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.092387 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:37.092550 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.092409 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:37.092550 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:19:37.092499 4290 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-serving-cert: secret "maas-api-e2e-worker-w1-9855ff-serving-cert" not found Nov 03 15:19:37.092752 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:19:37.092569 4290 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-maas-api-tls podName:4fbf376d-4692-4f6c-9b1b-c571c2e44a2b nodeName:}" failed. No retries permitted until 2026-11-03 15:19:37.592548074 +0000 UTC m=+685.023505884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-maas-api-tls") pod "maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" (UID: "4fbf376d-4692-4f6c-9b1b-c571c2e44a2b") : secret "maas-api-e2e-worker-w1-9855ff-serving-cert" not found Nov 03 15:19:37.092752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.092586 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2nmj\" (UniqueName: \"kubernetes.io/projected/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-kube-api-access-w2nmj\") pod \"maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:37.093030 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.093009 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:37.095139 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.095112 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-metrics-certs\") pod \"maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:37.155992 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.155957 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2nmj\" (UniqueName: \"kubernetes.io/projected/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-kube-api-access-w2nmj\") pod \"maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:37.597406 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.597372 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:37.600038 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.600013 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:37.827013 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:37.826982 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:38.209181 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:38.209159 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9"] Nov 03 15:19:38.211675 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:19:38.211650 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbf376d_4692_4f6c_9b1b_c571c2e44a2b.slice/crio-6e75bdcbe670cb13044d78fc4309c5aac5d512a3cef3540098081da490296f0e WatchSource:0}: Error finding container 6e75bdcbe670cb13044d78fc4309c5aac5d512a3cef3540098081da490296f0e: Status 404 returned error can't find the container with id 6e75bdcbe670cb13044d78fc4309c5aac5d512a3cef3540098081da490296f0e Nov 03 15:19:38.237314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:38.237287 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" event={"ID":"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b","Type":"ContainerStarted","Data":"6e75bdcbe670cb13044d78fc4309c5aac5d512a3cef3540098081da490296f0e"} Nov 03 15:19:38.701411 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:19:38.701371 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:19:39.073479 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.073402 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw"] Nov 03 15:19:39.078721 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.078681 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" Nov 03 15:19:39.110535 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.110507 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c74b1f4-9c5f-4909-bb4e-92181b087fb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw\" (UID: \"7c74b1f4-9c5f-4909-bb4e-92181b087fb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" Nov 03 15:19:39.110783 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.110549 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpf25\" (UniqueName: \"kubernetes.io/projected/7c74b1f4-9c5f-4909-bb4e-92181b087fb2-kube-api-access-mpf25\") pod \"payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw\" (UID: \"7c74b1f4-9c5f-4909-bb4e-92181b087fb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" Nov 03 15:19:39.133164 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.133126 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-f59d26\"" Nov 03 15:19:39.136931 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.136909 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw"] Nov 03 15:19:39.184307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.184280 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-f59d26-dockercfg-s8xhm\"" Nov 03 15:19:39.211522 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.211489 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c74b1f4-9c5f-4909-bb4e-92181b087fb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw\" (UID: \"7c74b1f4-9c5f-4909-bb4e-92181b087fb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" Nov 03 15:19:39.211741 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.211554 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpf25\" (UniqueName: \"kubernetes.io/projected/7c74b1f4-9c5f-4909-bb4e-92181b087fb2-kube-api-access-mpf25\") pod \"payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw\" (UID: \"7c74b1f4-9c5f-4909-bb4e-92181b087fb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" Nov 03 15:19:39.212629 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.212599 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c74b1f4-9c5f-4909-bb4e-92181b087fb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw\" (UID: \"7c74b1f4-9c5f-4909-bb4e-92181b087fb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" Nov 03 15:19:39.238955 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.238924 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn"] Nov 03 15:19:39.243048 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.243026 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" Nov 03 15:19:39.274368 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.274339 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpf25\" (UniqueName: \"kubernetes.io/projected/7c74b1f4-9c5f-4909-bb4e-92181b087fb2-kube-api-access-mpf25\") pod \"payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw\" (UID: \"7c74b1f4-9c5f-4909-bb4e-92181b087fb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" Nov 03 15:19:39.286174 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.286137 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn"] Nov 03 15:19:39.312303 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.312275 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2nzl\" (UniqueName: \"kubernetes.io/projected/b8671d1f-ea5b-4433-af1d-40b53480728b-kube-api-access-k2nzl\") pod \"payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn\" (UID: \"b8671d1f-ea5b-4433-af1d-40b53480728b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" Nov 03 15:19:39.312466 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.312448 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8671d1f-ea5b-4433-af1d-40b53480728b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn\" (UID: \"b8671d1f-ea5b-4433-af1d-40b53480728b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" Nov 03 15:19:39.392522 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.392487 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" Nov 03 15:19:39.413529 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.413498 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8671d1f-ea5b-4433-af1d-40b53480728b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn\" (UID: \"b8671d1f-ea5b-4433-af1d-40b53480728b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" Nov 03 15:19:39.413693 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.413540 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2nzl\" (UniqueName: \"kubernetes.io/projected/b8671d1f-ea5b-4433-af1d-40b53480728b-kube-api-access-k2nzl\") pod \"payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn\" (UID: \"b8671d1f-ea5b-4433-af1d-40b53480728b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" Nov 03 15:19:39.414555 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.414528 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8671d1f-ea5b-4433-af1d-40b53480728b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn\" (UID: \"b8671d1f-ea5b-4433-af1d-40b53480728b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" Nov 03 15:19:39.482600 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.482569 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2nzl\" (UniqueName: \"kubernetes.io/projected/b8671d1f-ea5b-4433-af1d-40b53480728b-kube-api-access-k2nzl\") pod \"payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn\" (UID: \"b8671d1f-ea5b-4433-af1d-40b53480728b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" Nov 03 15:19:39.556872 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.556769 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" Nov 03 15:19:39.786137 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.782939 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw"] Nov 03 15:19:39.946152 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:39.946129 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn"] Nov 03 15:19:39.949003 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:19:39.948972 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8671d1f_ea5b_4433_af1d_40b53480728b.slice/crio-572f285ba07f23b95305bb69c12874ad11fb2988df2e51fb0a2faf75fbe3b6b4 WatchSource:0}: Error finding container 572f285ba07f23b95305bb69c12874ad11fb2988df2e51fb0a2faf75fbe3b6b4: Status 404 returned error can't find the container with id 572f285ba07f23b95305bb69c12874ad11fb2988df2e51fb0a2faf75fbe3b6b4 Nov 03 15:19:40.253206 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:40.253166 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" event={"ID":"7c74b1f4-9c5f-4909-bb4e-92181b087fb2","Type":"ContainerStarted","Data":"fbef979bc24746a8e15b5f8fc073c1f5ddd280b1e9452b4ccf951736467eda67"} Nov 03 15:19:40.253392 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:40.253234 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" event={"ID":"7c74b1f4-9c5f-4909-bb4e-92181b087fb2","Type":"ContainerStarted","Data":"b49446805d83fed235db9c709f874ad5264968f35ec65813b6615bff237e77b9"} Nov 03 15:19:40.253473 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:40.253391 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" Nov 03 15:19:40.255364 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:40.255338 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" event={"ID":"b8671d1f-ea5b-4433-af1d-40b53480728b","Type":"ContainerStarted","Data":"a975247eb2830e50582e3a1e29cd375853ee5c2c5782436473ee8d6f2790d0d4"} Nov 03 15:19:40.255506 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:40.255384 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" event={"ID":"b8671d1f-ea5b-4433-af1d-40b53480728b","Type":"ContainerStarted","Data":"572f285ba07f23b95305bb69c12874ad11fb2988df2e51fb0a2faf75fbe3b6b4"} Nov 03 15:19:40.255506 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:40.255473 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" Nov 03 15:19:40.374336 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:40.374285 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" podStartSLOduration=1.374269494 podStartE2EDuration="1.374269494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:40.372026156 +0000 UTC m=+687.802983961" watchObservedRunningTime="2026-11-03 15:19:40.374269494 +0000 UTC m=+687.805227303" Nov 03 15:19:40.500499 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:40.500434 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" podStartSLOduration=1.5004123150000002 podStartE2EDuration="1.500412315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:40.495035625 +0000 UTC m=+687.925993429" watchObservedRunningTime="2026-11-03 15:19:40.500412315 +0000 UTC m=+687.931370121" Nov 03 15:19:41.272189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.271879 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" event={"ID":"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b","Type":"ContainerStarted","Data":"ce869874a9ca27e202cd44490ed67c0a34fc1f446cb89ef5f2c9f276d57e628f"} Nov 03 15:19:41.272189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.272041 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:41.403315 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.403253 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" podStartSLOduration=3.145376156 podStartE2EDuration="5.403235195s" totalImagesPullingTime="2.257859039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:41.403085261 +0000 UTC m=+688.834043069" watchObservedRunningTime="2026-11-03 15:19:41.403235195 +0000 UTC m=+688.834193000" Nov 03 15:19:41.602625 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.602540 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc"] Nov 03 15:19:41.606583 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.606562 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" Nov 03 15:19:41.636922 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.636898 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd9a4e78-c549-4d50-8b0a-05552100f91f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc\" (UID: \"bd9a4e78-c549-4d50-8b0a-05552100f91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" Nov 03 15:19:41.637047 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.636931 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vd4p\" (UniqueName: \"kubernetes.io/projected/bd9a4e78-c549-4d50-8b0a-05552100f91f-kube-api-access-9vd4p\") pod \"payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc\" (UID: \"bd9a4e78-c549-4d50-8b0a-05552100f91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" Nov 03 15:19:41.662971 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.662945 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-9855ff-dockercfg-8kpq7\"" Nov 03 15:19:41.663113 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.663007 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-9855ff\"" Nov 03 15:19:41.666417 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.666394 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc"] Nov 03 15:19:41.735948 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.735916 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d"] Nov 03 15:19:41.738480 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.738418 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd9a4e78-c549-4d50-8b0a-05552100f91f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc\" (UID: \"bd9a4e78-c549-4d50-8b0a-05552100f91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" Nov 03 15:19:41.738651 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.738498 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vd4p\" (UniqueName: \"kubernetes.io/projected/bd9a4e78-c549-4d50-8b0a-05552100f91f-kube-api-access-9vd4p\") pod \"payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc\" (UID: \"bd9a4e78-c549-4d50-8b0a-05552100f91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" Nov 03 15:19:41.739412 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.739391 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd9a4e78-c549-4d50-8b0a-05552100f91f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc\" (UID: \"bd9a4e78-c549-4d50-8b0a-05552100f91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" Nov 03 15:19:41.739923 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.739904 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" Nov 03 15:19:41.799011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.798977 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d"] Nov 03 15:19:41.806876 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.806848 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vd4p\" (UniqueName: \"kubernetes.io/projected/bd9a4e78-c549-4d50-8b0a-05552100f91f-kube-api-access-9vd4p\") pod \"payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc\" (UID: \"bd9a4e78-c549-4d50-8b0a-05552100f91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" Nov 03 15:19:41.840003 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.839957 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5vjb\" (UniqueName: \"kubernetes.io/projected/65b4e93f-0dce-4a32-9248-35979203ae23-kube-api-access-d5vjb\") pod \"payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d\" (UID: \"65b4e93f-0dce-4a32-9248-35979203ae23\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" Nov 03 15:19:41.840157 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.840114 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b4e93f-0dce-4a32-9248-35979203ae23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d\" (UID: \"65b4e93f-0dce-4a32-9248-35979203ae23\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" Nov 03 15:19:41.920223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.920193 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" Nov 03 15:19:41.941036 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.940810 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b4e93f-0dce-4a32-9248-35979203ae23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d\" (UID: \"65b4e93f-0dce-4a32-9248-35979203ae23\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" Nov 03 15:19:41.941036 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.940990 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vjb\" (UniqueName: \"kubernetes.io/projected/65b4e93f-0dce-4a32-9248-35979203ae23-kube-api-access-d5vjb\") pod \"payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d\" (UID: \"65b4e93f-0dce-4a32-9248-35979203ae23\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" Nov 03 15:19:41.942191 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:41.942168 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b4e93f-0dce-4a32-9248-35979203ae23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d\" (UID: \"65b4e93f-0dce-4a32-9248-35979203ae23\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" Nov 03 15:19:42.014166 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.014105 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vjb\" (UniqueName: \"kubernetes.io/projected/65b4e93f-0dce-4a32-9248-35979203ae23-kube-api-access-d5vjb\") pod \"payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d\" (UID: \"65b4e93f-0dce-4a32-9248-35979203ae23\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" Nov 03 15:19:42.053296 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.053263 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" Nov 03 15:19:42.313454 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.313366 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc"] Nov 03 15:19:42.317617 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.317587 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2"] Nov 03 15:19:42.319131 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:19:42.319106 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9a4e78_c549_4d50_8b0a_05552100f91f.slice/crio-684e43adde07e6072086c76cc35a3e63fc27b5a1bfbb520eb3656098f17f1a7e WatchSource:0}: Error finding container 684e43adde07e6072086c76cc35a3e63fc27b5a1bfbb520eb3656098f17f1a7e: Status 404 returned error can't find the container with id 684e43adde07e6072086c76cc35a3e63fc27b5a1bfbb520eb3656098f17f1a7e Nov 03 15:19:42.323400 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.323376 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:42.345515 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.345491 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqnvz\" (UniqueName: \"kubernetes.io/projected/ce2f1c46-2aae-4048-8008-105f58fceffe-kube-api-access-mqnvz\") pod \"maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:42.345641 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.345551 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-metrics-certs\") pod \"maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:42.345794 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.345772 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2f1c46-2aae-4048-8008-105f58fceffe-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:42.345870 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.345822 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-maas-api-tls\") pod \"maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:42.377976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.377945 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-34e8c5-serving-cert\"" Nov 03 15:19:42.424752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.424719 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2"] Nov 03 15:19:42.435977 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.435936 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d"] Nov 03 15:19:42.438777 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:19:42.438748 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b4e93f_0dce_4a32_9248_35979203ae23.slice/crio-523489e0a87d7f68b5dc4c59ee75f0b882538778911dd1e17764aa1ba60caf2c WatchSource:0}: Error finding container 523489e0a87d7f68b5dc4c59ee75f0b882538778911dd1e17764aa1ba60caf2c: Status 404 returned error can't find the container with id 523489e0a87d7f68b5dc4c59ee75f0b882538778911dd1e17764aa1ba60caf2c Nov 03 15:19:42.447016 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.446987 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqnvz\" (UniqueName: \"kubernetes.io/projected/ce2f1c46-2aae-4048-8008-105f58fceffe-kube-api-access-mqnvz\") pod \"maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:42.447152 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.447058 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-metrics-certs\") pod \"maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:42.447152 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.447112 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2f1c46-2aae-4048-8008-105f58fceffe-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:42.447272 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.447150 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-maas-api-tls\") pod \"maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:42.447326 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:19:42.447282 4290 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-serving-cert: secret "maas-api-e2e-worker-w5-34e8c5-serving-cert" not found Nov 03 15:19:42.447397 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:19:42.447339 4290 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-maas-api-tls podName:ce2f1c46-2aae-4048-8008-105f58fceffe nodeName:}" failed. No retries permitted until 2026-11-03 15:19:42.947319551 +0000 UTC m=+690.378277350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-maas-api-tls") pod "maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" (UID: "ce2f1c46-2aae-4048-8008-105f58fceffe") : secret "maas-api-e2e-worker-w5-34e8c5-serving-cert" not found Nov 03 15:19:42.448043 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.447996 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2f1c46-2aae-4048-8008-105f58fceffe-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:42.450204 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.450183 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-metrics-certs\") pod \"maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:42.514326 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.514301 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqnvz\" (UniqueName: \"kubernetes.io/projected/ce2f1c46-2aae-4048-8008-105f58fceffe-kube-api-access-mqnvz\") pod \"maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:42.952991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.952949 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-maas-api-tls\") pod \"maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:42.955783 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:42.955754 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-maas-api-tls\") pod \"maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:43.247194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:43.247088 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:43.287164 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:43.287130 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" event={"ID":"bd9a4e78-c549-4d50-8b0a-05552100f91f","Type":"ContainerStarted","Data":"cfca423a50b3380e76176299a611ea82331250ad443b8bed91b5a780cb263f38"} Nov 03 15:19:43.287323 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:43.287196 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" Nov 03 15:19:43.287323 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:43.287218 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" event={"ID":"bd9a4e78-c549-4d50-8b0a-05552100f91f","Type":"ContainerStarted","Data":"684e43adde07e6072086c76cc35a3e63fc27b5a1bfbb520eb3656098f17f1a7e"} Nov 03 15:19:43.288973 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:43.288943 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" event={"ID":"65b4e93f-0dce-4a32-9248-35979203ae23","Type":"ContainerStarted","Data":"81bce026ba38af283bc21060ea75b21f7a3986e4915bbc7a4c02790cf2bbff8b"} Nov 03 15:19:43.289113 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:43.288992 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" event={"ID":"65b4e93f-0dce-4a32-9248-35979203ae23","Type":"ContainerStarted","Data":"523489e0a87d7f68b5dc4c59ee75f0b882538778911dd1e17764aa1ba60caf2c"} Nov 03 15:19:43.289169 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:43.289122 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" Nov 03 15:19:43.409926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:43.409876 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" podStartSLOduration=2.409863246 podStartE2EDuration="2.409863246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:43.406427245 +0000 UTC m=+690.837385051" watchObservedRunningTime="2026-11-03 15:19:43.409863246 +0000 UTC m=+690.840821052" Nov 03 15:19:43.529511 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:43.529338 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" podStartSLOduration=2.529321489 podStartE2EDuration="2.529321489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:43.52493983 +0000 UTC m=+690.955897636" watchObservedRunningTime="2026-11-03 15:19:43.529321489 +0000 UTC m=+690.960279293" Nov 03 15:19:43.693890 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:43.693857 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2"] Nov 03 15:19:43.696401 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:19:43.696373 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2f1c46_2aae_4048_8008_105f58fceffe.slice/crio-3112e73167a6d9f5da4313f2181b2a5450b1ae365714e10634ae076bc6bc6a81 WatchSource:0}: Error finding container 3112e73167a6d9f5da4313f2181b2a5450b1ae365714e10634ae076bc6bc6a81: Status 404 returned error can't find the container with id 3112e73167a6d9f5da4313f2181b2a5450b1ae365714e10634ae076bc6bc6a81 Nov 03 15:19:44.199513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.199478 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8"] Nov 03 15:19:44.204774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.204752 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" Nov 03 15:19:44.263587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.263560 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8"] Nov 03 15:19:44.267221 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.267198 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tzhh\" (UniqueName: \"kubernetes.io/projected/0866b376-9269-41cf-aa1a-d29d433718e7-kube-api-access-4tzhh\") pod \"payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8\" (UID: \"0866b376-9269-41cf-aa1a-d29d433718e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" Nov 03 15:19:44.267344 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.267247 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0866b376-9269-41cf-aa1a-d29d433718e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8\" (UID: \"0866b376-9269-41cf-aa1a-d29d433718e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" Nov 03 15:19:44.303722 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.302857 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" event={"ID":"ce2f1c46-2aae-4048-8008-105f58fceffe","Type":"ContainerStarted","Data":"3112e73167a6d9f5da4313f2181b2a5450b1ae365714e10634ae076bc6bc6a81"} Nov 03 15:19:44.311048 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.311019 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-8b841a-dockercfg-tnhg5\"" Nov 03 15:19:44.311234 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.311085 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-8b841a\"" Nov 03 15:19:44.348319 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.346454 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d"] Nov 03 15:19:44.351178 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.351154 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" Nov 03 15:19:44.368048 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.368021 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tzhh\" (UniqueName: \"kubernetes.io/projected/0866b376-9269-41cf-aa1a-d29d433718e7-kube-api-access-4tzhh\") pod \"payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8\" (UID: \"0866b376-9269-41cf-aa1a-d29d433718e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" Nov 03 15:19:44.368168 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.368099 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0866b376-9269-41cf-aa1a-d29d433718e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8\" (UID: \"0866b376-9269-41cf-aa1a-d29d433718e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" Nov 03 15:19:44.368168 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.368138 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d\" (UID: \"d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" Nov 03 15:19:44.368274 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.368212 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxxpt\" (UniqueName: \"kubernetes.io/projected/d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36-kube-api-access-lxxpt\") pod \"payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d\" (UID: \"d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" Nov 03 15:19:44.368783 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.368765 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0866b376-9269-41cf-aa1a-d29d433718e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8\" (UID: \"0866b376-9269-41cf-aa1a-d29d433718e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" Nov 03 15:19:44.405412 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.405379 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d"] Nov 03 15:19:44.431167 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.431143 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tzhh\" (UniqueName: \"kubernetes.io/projected/0866b376-9269-41cf-aa1a-d29d433718e7-kube-api-access-4tzhh\") pod \"payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8\" (UID: \"0866b376-9269-41cf-aa1a-d29d433718e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" Nov 03 15:19:44.469205 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.469103 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d\" (UID: \"d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" Nov 03 15:19:44.469205 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.469199 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxpt\" (UniqueName: \"kubernetes.io/projected/d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36-kube-api-access-lxxpt\") pod \"payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d\" (UID: \"d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" Nov 03 15:19:44.470043 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.470020 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d\" (UID: \"d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" Nov 03 15:19:44.518376 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.518353 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" Nov 03 15:19:44.532269 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.532244 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxxpt\" (UniqueName: \"kubernetes.io/projected/d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36-kube-api-access-lxxpt\") pod \"payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d\" (UID: \"d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" Nov 03 15:19:44.664845 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.664820 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" Nov 03 15:19:44.901515 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:44.901492 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8"] Nov 03 15:19:44.904026 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:19:44.903993 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0866b376_9269_41cf_aa1a_d29d433718e7.slice/crio-8061d19a3a259c32af693896005dd378139ad60b8f5f3f4280de68a5a0633159 WatchSource:0}: Error finding container 8061d19a3a259c32af693896005dd378139ad60b8f5f3f4280de68a5a0633159: Status 404 returned error can't find the container with id 8061d19a3a259c32af693896005dd378139ad60b8f5f3f4280de68a5a0633159 Nov 03 15:19:45.055539 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:45.055509 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d"] Nov 03 15:19:45.056598 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:19:45.056568 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c7a85c_0cc1_4dcb_ae0d_45ea7a625c36.slice/crio-c1075121c351eb0f4bfb6ec043fd5e4e27f2c001ac35fef32549f71a9f832556 WatchSource:0}: Error finding container c1075121c351eb0f4bfb6ec043fd5e4e27f2c001ac35fef32549f71a9f832556: Status 404 returned error can't find the container with id c1075121c351eb0f4bfb6ec043fd5e4e27f2c001ac35fef32549f71a9f832556 Nov 03 15:19:45.312661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:45.312558 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" event={"ID":"ce2f1c46-2aae-4048-8008-105f58fceffe","Type":"ContainerStarted","Data":"aeee40f2888f9ebb075f9c4c268c4e0e76206a88fbfe4bc49937f5267689c26d"} Nov 03 15:19:45.312661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:45.312633 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:45.314444 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:45.314416 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" event={"ID":"d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36","Type":"ContainerStarted","Data":"0dbdb933f7faec5c73a8f35b077def64251dcc0b7bcfc7002b0cb136eec89ff0"} Nov 03 15:19:45.314555 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:45.314468 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" event={"ID":"d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36","Type":"ContainerStarted","Data":"c1075121c351eb0f4bfb6ec043fd5e4e27f2c001ac35fef32549f71a9f832556"} Nov 03 15:19:45.314555 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:45.314501 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" Nov 03 15:19:45.316253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:45.316228 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" event={"ID":"0866b376-9269-41cf-aa1a-d29d433718e7","Type":"ContainerStarted","Data":"f2bfaff1cff0fb968300f103fa969cf5bf04be032eab2331a9f2404e2ff05310"} Nov 03 15:19:45.316327 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:45.316269 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" event={"ID":"0866b376-9269-41cf-aa1a-d29d433718e7","Type":"ContainerStarted","Data":"8061d19a3a259c32af693896005dd378139ad60b8f5f3f4280de68a5a0633159"} Nov 03 15:19:45.316327 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:45.316288 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" Nov 03 15:19:45.435214 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:45.435163 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" podStartSLOduration=2.470231052 podStartE2EDuration="3.43514748s" totalImagesPullingTime="964.916428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:45.434178516 +0000 UTC m=+692.865136322" watchObservedRunningTime="2026-11-03 15:19:45.43514748 +0000 UTC m=+692.866105285" Nov 03 15:19:45.694074 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:45.694026 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" podStartSLOduration=1.6940071859999999 podStartE2EDuration="1.694007186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:45.559882708 +0000 UTC m=+692.990840512" watchObservedRunningTime="2026-11-03 15:19:45.694007186 +0000 UTC m=+693.124964995" Nov 03 15:19:45.694292 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:45.694125 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" podStartSLOduration=1.694118183 podStartE2EDuration="1.694118183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:45.690587012 +0000 UTC m=+693.121544819" watchObservedRunningTime="2026-11-03 15:19:45.694118183 +0000 UTC m=+693.125075990" Nov 03 15:19:46.227077 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:46.227047 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" Nov 03 15:19:47.233808 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:47.233778 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" Nov 03 15:19:48.296244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:48.296214 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:19:51.275792 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:51.275762 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" Nov 03 15:19:51.276287 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:51.275842 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" Nov 03 15:19:51.328770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:51.328739 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:19:53.675211 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.675167 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn"] Nov 03 15:19:53.680438 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.680415 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" Nov 03 15:19:53.742522 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.742492 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn"] Nov 03 15:19:53.767316 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.767272 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0000e843-b56d-4f34-8001-65de8a8faaa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn\" (UID: \"0000e843-b56d-4f34-8001-65de8a8faaa8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" Nov 03 15:19:53.767503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.767355 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8js5\" (UniqueName: \"kubernetes.io/projected/0000e843-b56d-4f34-8001-65de8a8faaa8-kube-api-access-n8js5\") pod \"payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn\" (UID: \"0000e843-b56d-4f34-8001-65de8a8faaa8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" Nov 03 15:19:53.789961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.789926 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-f78fd6\"" Nov 03 15:19:53.789961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.789932 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-f78fd6-dockercfg-wp675\"" Nov 03 15:19:53.848724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.845155 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x"] Nov 03 15:19:53.855888 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.851478 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" Nov 03 15:19:53.868274 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.868245 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8js5\" (UniqueName: \"kubernetes.io/projected/0000e843-b56d-4f34-8001-65de8a8faaa8-kube-api-access-n8js5\") pod \"payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn\" (UID: \"0000e843-b56d-4f34-8001-65de8a8faaa8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" Nov 03 15:19:53.868450 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.868434 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0000e843-b56d-4f34-8001-65de8a8faaa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn\" (UID: \"0000e843-b56d-4f34-8001-65de8a8faaa8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" Nov 03 15:19:53.869068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.869050 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0000e843-b56d-4f34-8001-65de8a8faaa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn\" (UID: \"0000e843-b56d-4f34-8001-65de8a8faaa8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" Nov 03 15:19:53.887353 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.887324 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x"] Nov 03 15:19:53.931761 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.931659 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8js5\" (UniqueName: \"kubernetes.io/projected/0000e843-b56d-4f34-8001-65de8a8faaa8-kube-api-access-n8js5\") pod \"payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn\" (UID: \"0000e843-b56d-4f34-8001-65de8a8faaa8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" Nov 03 15:19:53.969111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.969081 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzrld\" (UniqueName: \"kubernetes.io/projected/c05e39ac-2a6d-4071-887b-ee32f91514a9-kube-api-access-zzrld\") pod \"payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x\" (UID: \"c05e39ac-2a6d-4071-887b-ee32f91514a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" Nov 03 15:19:53.969272 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.969246 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05e39ac-2a6d-4071-887b-ee32f91514a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x\" (UID: \"c05e39ac-2a6d-4071-887b-ee32f91514a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" Nov 03 15:19:53.995445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:53.995414 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" Nov 03 15:19:54.072395 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:54.070577 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05e39ac-2a6d-4071-887b-ee32f91514a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x\" (UID: \"c05e39ac-2a6d-4071-887b-ee32f91514a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" Nov 03 15:19:54.072395 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:54.071822 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05e39ac-2a6d-4071-887b-ee32f91514a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x\" (UID: \"c05e39ac-2a6d-4071-887b-ee32f91514a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" Nov 03 15:19:54.072395 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:54.071861 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzrld\" (UniqueName: \"kubernetes.io/projected/c05e39ac-2a6d-4071-887b-ee32f91514a9-kube-api-access-zzrld\") pod \"payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x\" (UID: \"c05e39ac-2a6d-4071-887b-ee32f91514a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" Nov 03 15:19:54.133641 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:54.133605 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzrld\" (UniqueName: \"kubernetes.io/projected/c05e39ac-2a6d-4071-887b-ee32f91514a9-kube-api-access-zzrld\") pod \"payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x\" (UID: \"c05e39ac-2a6d-4071-887b-ee32f91514a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" Nov 03 15:19:54.169683 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:54.169633 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" Nov 03 15:19:54.306407 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:54.306370 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" Nov 03 15:19:54.306672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:54.306652 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" Nov 03 15:19:54.378927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:54.378903 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn"] Nov 03 15:19:54.380971 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:19:54.380947 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0000e843_b56d_4f34_8001_65de8a8faaa8.slice/crio-edbb559441449df79f6a126e86e94717bffe18c22910a33d453e03583c6d04ee WatchSource:0}: Error finding container edbb559441449df79f6a126e86e94717bffe18c22910a33d453e03583c6d04ee: Status 404 returned error can't find the container with id edbb559441449df79f6a126e86e94717bffe18c22910a33d453e03583c6d04ee Nov 03 15:19:54.391556 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:54.391527 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" event={"ID":"0000e843-b56d-4f34-8001-65de8a8faaa8","Type":"ContainerStarted","Data":"edbb559441449df79f6a126e86e94717bffe18c22910a33d453e03583c6d04ee"} Nov 03 15:19:54.575738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:54.571796 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x"] Nov 03 15:19:54.578019 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:19:54.577979 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05e39ac_2a6d_4071_887b_ee32f91514a9.slice/crio-d589ab679ef5b0d095067c6554f9536aba1e9c8ca3fcd6208510c6013e645a23 WatchSource:0}: Error finding container d589ab679ef5b0d095067c6554f9536aba1e9c8ca3fcd6208510c6013e645a23: Status 404 returned error can't find the container with id d589ab679ef5b0d095067c6554f9536aba1e9c8ca3fcd6208510c6013e645a23 Nov 03 15:19:55.399313 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:55.399276 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" event={"ID":"c05e39ac-2a6d-4071-887b-ee32f91514a9","Type":"ContainerStarted","Data":"53d77dc8d3e6713526aa91a0ca00333ff616fe8ec485ea2ea8426c72e85d69a1"} Nov 03 15:19:55.399799 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:55.399336 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" event={"ID":"c05e39ac-2a6d-4071-887b-ee32f91514a9","Type":"ContainerStarted","Data":"d589ab679ef5b0d095067c6554f9536aba1e9c8ca3fcd6208510c6013e645a23"} Nov 03 15:19:55.399799 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:55.399364 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" Nov 03 15:19:55.401024 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:55.400992 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" event={"ID":"0000e843-b56d-4f34-8001-65de8a8faaa8","Type":"ContainerStarted","Data":"97b4781aa30210e5606ba726b42dfbd817e5f17de18f23b1d163fa3bf24a7d38"} Nov 03 15:19:55.401134 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:55.401036 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" Nov 03 15:19:55.573101 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:55.573056 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" podStartSLOduration=2.57304137 podStartE2EDuration="2.57304137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:55.518902755 +0000 UTC m=+702.949860561" watchObservedRunningTime="2026-11-03 15:19:55.57304137 +0000 UTC m=+703.003999240" Nov 03 15:19:55.639690 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:55.639642 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" podStartSLOduration=2.639628955 podStartE2EDuration="2.639628955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:55.639209162 +0000 UTC m=+703.070166969" watchObservedRunningTime="2026-11-03 15:19:55.639628955 +0000 UTC m=+703.070586802" Nov 03 15:19:56.244886 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.244850 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft"] Nov 03 15:19:56.249077 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.249056 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" Nov 03 15:19:56.302330 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.302293 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce34d51-acb2-484d-acee-275b15ec6532-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft\" (UID: \"5ce34d51-acb2-484d-acee-275b15ec6532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" Nov 03 15:19:56.302496 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.302421 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twmwh\" (UniqueName: \"kubernetes.io/projected/5ce34d51-acb2-484d-acee-275b15ec6532-kube-api-access-twmwh\") pod \"payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft\" (UID: \"5ce34d51-acb2-484d-acee-275b15ec6532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" Nov 03 15:19:56.311828 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.311795 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft"] Nov 03 15:19:56.338172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.337963 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" Nov 03 15:19:56.339064 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.339044 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" Nov 03 15:19:56.356781 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.356748 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-bc1f07\"" Nov 03 15:19:56.356781 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.356750 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-bc1f07-dockercfg-rgb9d\"" Nov 03 15:19:56.397004 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.396974 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t"] Nov 03 15:19:56.401130 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.401106 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" Nov 03 15:19:56.403223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.403191 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce34d51-acb2-484d-acee-275b15ec6532-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft\" (UID: \"5ce34d51-acb2-484d-acee-275b15ec6532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" Nov 03 15:19:56.403361 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.403328 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twmwh\" (UniqueName: \"kubernetes.io/projected/5ce34d51-acb2-484d-acee-275b15ec6532-kube-api-access-twmwh\") pod \"payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft\" (UID: \"5ce34d51-acb2-484d-acee-275b15ec6532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" Nov 03 15:19:56.404056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.404031 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce34d51-acb2-484d-acee-275b15ec6532-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft\" (UID: \"5ce34d51-acb2-484d-acee-275b15ec6532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" Nov 03 15:19:56.463287 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.463001 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t"] Nov 03 15:19:56.468266 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.468235 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twmwh\" (UniqueName: \"kubernetes.io/projected/5ce34d51-acb2-484d-acee-275b15ec6532-kube-api-access-twmwh\") pod \"payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft\" (UID: \"5ce34d51-acb2-484d-acee-275b15ec6532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" Nov 03 15:19:56.504171 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.504092 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3f085b-e44a-4f7a-a92e-065c19adf7d8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t\" (UID: \"ad3f085b-e44a-4f7a-a92e-065c19adf7d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" Nov 03 15:19:56.504171 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.504145 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvjrw\" (UniqueName: \"kubernetes.io/projected/ad3f085b-e44a-4f7a-a92e-065c19adf7d8-kube-api-access-qvjrw\") pod \"payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t\" (UID: \"ad3f085b-e44a-4f7a-a92e-065c19adf7d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" Nov 03 15:19:56.563966 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.563933 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" Nov 03 15:19:56.605797 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.605758 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3f085b-e44a-4f7a-a92e-065c19adf7d8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t\" (UID: \"ad3f085b-e44a-4f7a-a92e-065c19adf7d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" Nov 03 15:19:56.605980 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.605826 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvjrw\" (UniqueName: \"kubernetes.io/projected/ad3f085b-e44a-4f7a-a92e-065c19adf7d8-kube-api-access-qvjrw\") pod \"payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t\" (UID: \"ad3f085b-e44a-4f7a-a92e-065c19adf7d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" Nov 03 15:19:56.606537 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.606511 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3f085b-e44a-4f7a-a92e-065c19adf7d8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t\" (UID: \"ad3f085b-e44a-4f7a-a92e-065c19adf7d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" Nov 03 15:19:56.669442 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.669411 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvjrw\" (UniqueName: \"kubernetes.io/projected/ad3f085b-e44a-4f7a-a92e-065c19adf7d8-kube-api-access-qvjrw\") pod \"payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t\" (UID: \"ad3f085b-e44a-4f7a-a92e-065c19adf7d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" Nov 03 15:19:56.716374 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.716343 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" Nov 03 15:19:56.951403 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:56.951379 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft"] Nov 03 15:19:56.954153 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:19:56.954126 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce34d51_acb2_484d_acee_275b15ec6532.slice/crio-be917eae63db53cdec1c556bf0d5a4e1b316bceb88c790e613cc3b1009e4d6d1 WatchSource:0}: Error finding container be917eae63db53cdec1c556bf0d5a4e1b316bceb88c790e613cc3b1009e4d6d1: Status 404 returned error can't find the container with id be917eae63db53cdec1c556bf0d5a4e1b316bceb88c790e613cc3b1009e4d6d1 Nov 03 15:19:57.104825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:57.104798 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t"] Nov 03 15:19:57.107337 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:19:57.107312 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3f085b_e44a_4f7a_a92e_065c19adf7d8.slice/crio-4d3cd855d557b38490a53e8f164d008f5912970f42fd53e1e77aaec83645e86f WatchSource:0}: Error finding container 4d3cd855d557b38490a53e8f164d008f5912970f42fd53e1e77aaec83645e86f: Status 404 returned error can't find the container with id 4d3cd855d557b38490a53e8f164d008f5912970f42fd53e1e77aaec83645e86f Nov 03 15:19:57.416508 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:57.416475 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" event={"ID":"ad3f085b-e44a-4f7a-a92e-065c19adf7d8","Type":"ContainerStarted","Data":"2eed4a4835c16f79ef0d8917c228f7939821e72dd69278a6a33230da3d11288e"} Nov 03 15:19:57.416973 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:57.416535 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" event={"ID":"ad3f085b-e44a-4f7a-a92e-065c19adf7d8","Type":"ContainerStarted","Data":"4d3cd855d557b38490a53e8f164d008f5912970f42fd53e1e77aaec83645e86f"} Nov 03 15:19:57.416973 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:57.416569 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" Nov 03 15:19:57.418227 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:57.418201 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" event={"ID":"5ce34d51-acb2-484d-acee-275b15ec6532","Type":"ContainerStarted","Data":"171fd0f269122493c3e40d6c80e6d504bc8d19d4f5439d998e5bfa112cf75c37"} Nov 03 15:19:57.418345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:57.418243 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" event={"ID":"5ce34d51-acb2-484d-acee-275b15ec6532","Type":"ContainerStarted","Data":"be917eae63db53cdec1c556bf0d5a4e1b316bceb88c790e613cc3b1009e4d6d1"} Nov 03 15:19:57.418345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:57.418268 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" Nov 03 15:19:57.542786 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:57.542728 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" podStartSLOduration=1.542687822 podStartE2EDuration="1.542687822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:57.542108538 +0000 UTC m=+704.973066348" watchObservedRunningTime="2026-11-03 15:19:57.542687822 +0000 UTC m=+704.973645672" Nov 03 15:19:57.662203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:57.662158 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" podStartSLOduration=1.662145301 podStartE2EDuration="1.662145301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:57.661951879 +0000 UTC m=+705.092909686" watchObservedRunningTime="2026-11-03 15:19:57.662145301 +0000 UTC m=+705.093103139" Nov 03 15:19:58.776245 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:58.776210 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt"] Nov 03 15:19:58.780500 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:58.780482 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" Nov 03 15:19:58.835682 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:58.831227 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e830bb-3e77-42b6-9d03-d41e24955b37-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt\" (UID: \"76e830bb-3e77-42b6-9d03-d41e24955b37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" Nov 03 15:19:58.835682 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:58.831294 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn5dq\" (UniqueName: \"kubernetes.io/projected/76e830bb-3e77-42b6-9d03-d41e24955b37-kube-api-access-jn5dq\") pod \"payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt\" (UID: \"76e830bb-3e77-42b6-9d03-d41e24955b37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" Nov 03 15:19:58.843016 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:58.842988 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt"] Nov 03 15:19:58.871206 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:58.871181 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e62ff3\"" Nov 03 15:19:58.887740 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:58.887677 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e62ff3-dockercfg-6z8ln\"" Nov 03 15:19:58.932436 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:58.932405 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e830bb-3e77-42b6-9d03-d41e24955b37-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt\" (UID: \"76e830bb-3e77-42b6-9d03-d41e24955b37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" Nov 03 15:19:58.932605 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:58.932446 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn5dq\" (UniqueName: \"kubernetes.io/projected/76e830bb-3e77-42b6-9d03-d41e24955b37-kube-api-access-jn5dq\") pod \"payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt\" (UID: \"76e830bb-3e77-42b6-9d03-d41e24955b37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" Nov 03 15:19:58.933143 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:58.933113 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e830bb-3e77-42b6-9d03-d41e24955b37-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt\" (UID: \"76e830bb-3e77-42b6-9d03-d41e24955b37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" Nov 03 15:19:58.995581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:58.995556 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn5dq\" (UniqueName: \"kubernetes.io/projected/76e830bb-3e77-42b6-9d03-d41e24955b37-kube-api-access-jn5dq\") pod \"payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt\" (UID: \"76e830bb-3e77-42b6-9d03-d41e24955b37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" Nov 03 15:19:59.095038 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:59.094962 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" Nov 03 15:19:59.478379 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:19:59.478355 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt"] Nov 03 15:19:59.481219 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:19:59.481184 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e830bb_3e77_42b6_9d03_d41e24955b37.slice/crio-26ae5e9dc94856f19f5962cc971b268b375c2f34e7526cc1c36d6fc4a908f59f WatchSource:0}: Error finding container 26ae5e9dc94856f19f5962cc971b268b375c2f34e7526cc1c36d6fc4a908f59f: Status 404 returned error can't find the container with id 26ae5e9dc94856f19f5962cc971b268b375c2f34e7526cc1c36d6fc4a908f59f Nov 03 15:20:00.442202 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:00.442168 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" event={"ID":"76e830bb-3e77-42b6-9d03-d41e24955b37","Type":"ContainerStarted","Data":"a9575c81007505760bcb470892761d230d58ebe5621e4581bfb881ac6312c156"} Nov 03 15:20:00.442753 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:00.442241 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" Nov 03 15:20:00.442753 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:00.442259 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" event={"ID":"76e830bb-3e77-42b6-9d03-d41e24955b37","Type":"ContainerStarted","Data":"26ae5e9dc94856f19f5962cc971b268b375c2f34e7526cc1c36d6fc4a908f59f"} Nov 03 15:20:00.564604 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:00.564550 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" podStartSLOduration=2.564537601 podStartE2EDuration="2.564537601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:20:00.560864948 +0000 UTC m=+707.991822752" watchObservedRunningTime="2026-11-03 15:20:00.564537601 +0000 UTC m=+707.995495405" Nov 03 15:20:06.411753 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:06.411722 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" Nov 03 15:20:06.412204 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:06.411921 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" Nov 03 15:20:08.430060 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.430032 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" Nov 03 15:20:08.430530 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.430104 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" Nov 03 15:20:08.638654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.638619 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt"] Nov 03 15:20:08.644042 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.644015 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:08.706502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.706418 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-e62ff3-serving-cert\"" Nov 03 15:20:08.712574 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.712541 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt"] Nov 03 15:20:08.717913 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:20:08.717881 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:20:08.736473 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.736423 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbq9\" (UniqueName: \"kubernetes.io/projected/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-kube-api-access-kdbq9\") pod \"maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:08.736614 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.736488 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:08.736678 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.736655 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-metrics-certs\") pod \"maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:08.736783 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.736765 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:08.837352 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.837321 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:08.837535 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.837416 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbq9\" (UniqueName: \"kubernetes.io/projected/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-kube-api-access-kdbq9\") pod \"maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:08.837535 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.837455 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:08.837535 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:20:08.837478 4290 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-serving-cert: secret "maas-api-e2e-worker-w0-e62ff3-serving-cert" not found Nov 03 15:20:08.837689 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:20:08.837561 4290 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-maas-api-tls podName:fbb5aae0-bbca-4f1f-83c3-875d0ffd9001 nodeName:}" failed. No retries permitted until 2026-11-03 15:20:09.337539707 +0000 UTC m=+716.768497511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-maas-api-tls") pod "maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" (UID: "fbb5aae0-bbca-4f1f-83c3-875d0ffd9001") : secret "maas-api-e2e-worker-w0-e62ff3-serving-cert" not found Nov 03 15:20:08.837689 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.837585 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-metrics-certs\") pod \"maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:08.838190 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.838162 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:08.840270 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.840247 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-metrics-certs\") pod \"maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:08.901527 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:08.901491 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbq9\" (UniqueName: \"kubernetes.io/projected/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-kube-api-access-kdbq9\") pod \"maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:09.343483 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:09.343447 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:09.346122 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:09.346093 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:09.558251 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:09.558220 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:09.950532 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:09.950507 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt"] Nov 03 15:20:09.953137 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:20:09.953109 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb5aae0_bbca_4f1f_83c3_875d0ffd9001.slice/crio-493c53f7a77ba5520bb836c156ffc73df02ef4bbf35932eba22a17e17c98c482 WatchSource:0}: Error finding container 493c53f7a77ba5520bb836c156ffc73df02ef4bbf35932eba22a17e17c98c482: Status 404 returned error can't find the container with id 493c53f7a77ba5520bb836c156ffc73df02ef4bbf35932eba22a17e17c98c482 Nov 03 15:20:10.517046 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:10.517001 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" event={"ID":"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001","Type":"ContainerStarted","Data":"493c53f7a77ba5520bb836c156ffc73df02ef4bbf35932eba22a17e17c98c482"} Nov 03 15:20:11.453265 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:11.453212 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" Nov 03 15:20:11.526849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:11.526817 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" event={"ID":"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001","Type":"ContainerStarted","Data":"c9f99df6f375629d00e5e0636c703b74935574979c5abbb36e44333b9d112162"} Nov 03 15:20:11.527053 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:11.526876 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:11.696972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:11.696879 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" podStartSLOduration=2.8984922749999997 podStartE2EDuration="3.696861466s" totalImagesPullingTime="798.369191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:20:11.695089732 +0000 UTC m=+719.126047539" watchObservedRunningTime="2026-11-03 15:20:11.696861466 +0000 UTC m=+719.127819272" Nov 03 15:20:17.538686 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:20:17.538658 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:20:38.702674 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:20:38.702631 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:21:08.704105 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:21:08.704022 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:21:38.704502 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:21:38.704456 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:22:08.704584 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:22:08.704495 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:22:35.544146 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.544064 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf"] Nov 03 15:22:35.548906 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.548884 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.652174 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.652123 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec99faf0-1759-4384-b30f-fb4c2679dfc4-tls-certs\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.652287 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.652182 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-model-cache\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.652349 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.652328 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqdbk\" (UniqueName: \"kubernetes.io/projected/ec99faf0-1759-4384-b30f-fb4c2679dfc4-kube-api-access-pqdbk\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.652402 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.652361 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-home\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.652511 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.652491 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-tmp-dir\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.652560 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.652534 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-dshm\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.654850 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.654829 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f78fd6\"/\"default-dockercfg-6sggv\"" Nov 03 15:22:35.654912 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.654866 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f78fd6\"/\"premium-w2-f78fd6-kserve-self-signed-certs\"" Nov 03 15:22:35.655040 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.655026 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f78fd6\"/\"kube-root-ca.crt\"" Nov 03 15:22:35.655154 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.655136 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f78fd6\"/\"openshift-service-ca.crt\"" Nov 03 15:22:35.658904 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.658882 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf"] Nov 03 15:22:35.753735 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.753682 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-home\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.753921 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.753831 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-tmp-dir\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.753921 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.753878 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-dshm\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.753921 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.753914 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec99faf0-1759-4384-b30f-fb4c2679dfc4-tls-certs\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.754100 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.753949 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-model-cache\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.754100 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.754056 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqdbk\" (UniqueName: \"kubernetes.io/projected/ec99faf0-1759-4384-b30f-fb4c2679dfc4-kube-api-access-pqdbk\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.754209 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.754118 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-home\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.754260 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.754209 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-tmp-dir\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.754368 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.754310 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-model-cache\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.756282 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.756257 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-dshm\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.756894 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.756873 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec99faf0-1759-4384-b30f-fb4c2679dfc4-tls-certs\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.820672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.820608 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqdbk\" (UniqueName: \"kubernetes.io/projected/ec99faf0-1759-4384-b30f-fb4c2679dfc4-kube-api-access-pqdbk\") pod \"premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:35.885792 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:35.885764 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:36.277799 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:36.277773 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf"] Nov 03 15:22:36.278825 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:22:36.278803 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec99faf0_1759_4384_b30f_fb4c2679dfc4.slice/crio-171a10ce935b0865efc78f33583440b213fb8d93950d56b99c2a5507b00ccf77 WatchSource:0}: Error finding container 171a10ce935b0865efc78f33583440b213fb8d93950d56b99c2a5507b00ccf77: Status 404 returned error can't find the container with id 171a10ce935b0865efc78f33583440b213fb8d93950d56b99c2a5507b00ccf77 Nov 03 15:22:36.566479 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:36.566388 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" event={"ID":"ec99faf0-1759-4384-b30f-fb4c2679dfc4","Type":"ContainerStarted","Data":"1c85c6e6264e4d72f5a52d0fe8a1c20cdec5c5d3743146736a3326dc37ed7ca4"} Nov 03 15:22:36.566479 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:36.566446 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" event={"ID":"ec99faf0-1759-4384-b30f-fb4c2679dfc4","Type":"ContainerStarted","Data":"171a10ce935b0865efc78f33583440b213fb8d93950d56b99c2a5507b00ccf77"} Nov 03 15:22:36.684095 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:36.684043 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podStartSLOduration=1.684027485 podStartE2EDuration="1.684027485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:22:36.683690904 +0000 UTC m=+864.114648723" watchObservedRunningTime="2026-11-03 15:22:36.684027485 +0000 UTC m=+864.114985291" Nov 03 15:22:36.886263 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:36.886235 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:38.702076 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:22:38.702043 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:22:45.886343 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:45.886307 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:45.899141 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:45.899110 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:22:46.650063 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:22:46.650037 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:23:05.303690 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.303660 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9"] Nov 03 15:23:05.309381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.308069 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.396813 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.396780 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-model-cache\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.397020 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.396820 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74jrn\" (UniqueName: \"kubernetes.io/projected/f90c3dc4-9f49-432a-9404-dd55360bcec0-kube-api-access-74jrn\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.397020 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.396920 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-tmp-dir\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.397020 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.396992 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f90c3dc4-9f49-432a-9404-dd55360bcec0-tls-certs\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.397020 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.397012 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-dshm\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.397201 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.397176 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-home\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.414444 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.414421 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f59d26\"/\"openshift-service-ca.crt\"" Nov 03 15:23:05.414444 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.414445 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f59d26\"/\"kube-root-ca.crt\"" Nov 03 15:23:05.414623 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.414586 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f59d26\"/\"default-dockercfg-7hssz\"" Nov 03 15:23:05.414674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.414662 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f59d26\"/\"premium-w6-f59d26-kserve-self-signed-certs\"" Nov 03 15:23:05.418181 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.418161 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9"] Nov 03 15:23:05.498614 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.498584 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f90c3dc4-9f49-432a-9404-dd55360bcec0-tls-certs\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.498614 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.498620 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-dshm\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.498855 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.498673 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-home\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.498925 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.498843 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-model-cache\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.498925 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.498891 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74jrn\" (UniqueName: \"kubernetes.io/projected/f90c3dc4-9f49-432a-9404-dd55360bcec0-kube-api-access-74jrn\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.499029 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.498950 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-tmp-dir\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.499095 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.499068 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-home\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.499197 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.499175 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-model-cache\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.499548 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.499429 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-tmp-dir\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.501125 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.501103 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-dshm\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.501232 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.501200 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f90c3dc4-9f49-432a-9404-dd55360bcec0-tls-certs\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.561122 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.561058 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74jrn\" (UniqueName: \"kubernetes.io/projected/f90c3dc4-9f49-432a-9404-dd55360bcec0-kube-api-access-74jrn\") pod \"premium-w6-f59d26-kserve-5b4df47dc6-c7pl9\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:05.626444 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:05.626413 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:06.019904 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:06.019879 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9"] Nov 03 15:23:06.020755 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:23:06.020733 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf90c3dc4_9f49_432a_9404_dd55360bcec0.slice/crio-033b102e148172b3e0ac5e0efcbdc1bb35864402f217cf164bd240624dd7a318 WatchSource:0}: Error finding container 033b102e148172b3e0ac5e0efcbdc1bb35864402f217cf164bd240624dd7a318: Status 404 returned error can't find the container with id 033b102e148172b3e0ac5e0efcbdc1bb35864402f217cf164bd240624dd7a318 Nov 03 15:23:06.786167 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:06.786135 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" event={"ID":"f90c3dc4-9f49-432a-9404-dd55360bcec0","Type":"ContainerStarted","Data":"d1c7076c686cbd27edfa7b1efe4570cd90f3f04d10973169ec5c6be2a534f41a"} Nov 03 15:23:06.786167 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:06.786173 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" event={"ID":"f90c3dc4-9f49-432a-9404-dd55360bcec0","Type":"ContainerStarted","Data":"033b102e148172b3e0ac5e0efcbdc1bb35864402f217cf164bd240624dd7a318"} Nov 03 15:23:06.907070 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:06.907014 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podStartSLOduration=1.906998922 podStartE2EDuration="1.906998922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:23:06.904603517 +0000 UTC m=+894.335561357" watchObservedRunningTime="2026-11-03 15:23:06.906998922 +0000 UTC m=+894.337956728" Nov 03 15:23:07.627136 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:07.627104 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:08.704356 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:23:08.704272 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:23:12.787507 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:12.787474 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:23:12.789796 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:12.789770 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:23:12.790319 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:12.790290 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:23:12.792513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:12.792495 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:23:15.627133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:15.627094 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:15.642237 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:15.642205 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:15.879348 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:23:15.879259 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:23:38.702003 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:23:38.701963 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:24:08.793855 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:24:08.793745 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:24:38.704588 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:24:38.704543 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:24:44.697782 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:44.697727 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r"] Nov 03 15:24:44.701958 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:44.701937 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" Nov 03 15:24:44.759781 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:44.759743 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r"] Nov 03 15:24:44.804211 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:44.804184 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:24:44.804371 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:44.804292 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:24:44.864667 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:44.864632 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc8sk\" (UniqueName: \"kubernetes.io/projected/332adcf3-f9d0-4c18-ab77-c358907288f3-kube-api-access-mc8sk\") pod \"maas-api-revoke-sub-d706c57c-gvw2r\" (UID: \"332adcf3-f9d0-4c18-ab77-c358907288f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" Nov 03 15:24:44.864870 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:44.864693 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/332adcf3-f9d0-4c18-ab77-c358907288f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d706c57c-gvw2r\" (UID: \"332adcf3-f9d0-4c18-ab77-c358907288f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" Nov 03 15:24:44.864937 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:44.864882 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/332adcf3-f9d0-4c18-ab77-c358907288f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d706c57c-gvw2r\" (UID: \"332adcf3-f9d0-4c18-ab77-c358907288f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" Nov 03 15:24:44.966284 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:44.966194 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8sk\" (UniqueName: \"kubernetes.io/projected/332adcf3-f9d0-4c18-ab77-c358907288f3-kube-api-access-mc8sk\") pod \"maas-api-revoke-sub-d706c57c-gvw2r\" (UID: \"332adcf3-f9d0-4c18-ab77-c358907288f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" Nov 03 15:24:44.966284 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:44.966244 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/332adcf3-f9d0-4c18-ab77-c358907288f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d706c57c-gvw2r\" (UID: \"332adcf3-f9d0-4c18-ab77-c358907288f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" Nov 03 15:24:44.966284 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:44.966281 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/332adcf3-f9d0-4c18-ab77-c358907288f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d706c57c-gvw2r\" (UID: \"332adcf3-f9d0-4c18-ab77-c358907288f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" Nov 03 15:24:44.966944 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:44.966920 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/332adcf3-f9d0-4c18-ab77-c358907288f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d706c57c-gvw2r\" (UID: \"332adcf3-f9d0-4c18-ab77-c358907288f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" Nov 03 15:24:44.968789 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:44.968766 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/332adcf3-f9d0-4c18-ab77-c358907288f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d706c57c-gvw2r\" (UID: \"332adcf3-f9d0-4c18-ab77-c358907288f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" Nov 03 15:24:45.028656 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:45.028621 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8sk\" (UniqueName: \"kubernetes.io/projected/332adcf3-f9d0-4c18-ab77-c358907288f3-kube-api-access-mc8sk\") pod \"maas-api-revoke-sub-d706c57c-gvw2r\" (UID: \"332adcf3-f9d0-4c18-ab77-c358907288f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" Nov 03 15:24:45.317560 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:45.317488 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" Nov 03 15:24:45.705354 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:45.705327 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r"] Nov 03 15:24:45.706467 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:24:45.706441 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332adcf3_f9d0_4c18_ab77_c358907288f3.slice/crio-260eb04caf63249b19842bf8305cba6281dd76412cd6f3faba7adc51f53c993d WatchSource:0}: Error finding container 260eb04caf63249b19842bf8305cba6281dd76412cd6f3faba7adc51f53c993d: Status 404 returned error can't find the container with id 260eb04caf63249b19842bf8305cba6281dd76412cd6f3faba7adc51f53c993d Nov 03 15:24:45.708443 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:45.708425 4290 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:24:46.666747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:24:46.666683 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" event={"ID":"332adcf3-f9d0-4c18-ab77-c358907288f3","Type":"ContainerStarted","Data":"260eb04caf63249b19842bf8305cba6281dd76412cd6f3faba7adc51f53c993d"} Nov 03 15:25:01.776115 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:01.776075 4290 generic.go:356] "Generic (PLEG): container finished" podID="332adcf3-f9d0-4c18-ab77-c358907288f3" containerID="2d2c71cee789e7298c772347f11c1a542cac6dd97b51c72a8c61e3a93dcb96fc" exitCode=0 Nov 03 15:25:01.776625 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:01.776205 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" event={"ID":"332adcf3-f9d0-4c18-ab77-c358907288f3","Type":"ContainerDied","Data":"2d2c71cee789e7298c772347f11c1a542cac6dd97b51c72a8c61e3a93dcb96fc"} Nov 03 15:25:02.939352 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:02.939326 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" Nov 03 15:25:03.081268 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:03.081172 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/332adcf3-f9d0-4c18-ab77-c358907288f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/332adcf3-f9d0-4c18-ab77-c358907288f3-maas-api-xks-ca\") pod \"332adcf3-f9d0-4c18-ab77-c358907288f3\" (UID: \"332adcf3-f9d0-4c18-ab77-c358907288f3\") " Nov 03 15:25:03.081268 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:03.081222 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332adcf3-f9d0-4c18-ab77-c358907288f3-kube-api-access-mc8sk\" (UniqueName: \"kubernetes.io/projected/332adcf3-f9d0-4c18-ab77-c358907288f3-kube-api-access-mc8sk\") pod \"332adcf3-f9d0-4c18-ab77-c358907288f3\" (UID: \"332adcf3-f9d0-4c18-ab77-c358907288f3\") " Nov 03 15:25:03.081500 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:03.081407 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/332adcf3-f9d0-4c18-ab77-c358907288f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/332adcf3-f9d0-4c18-ab77-c358907288f3-maas-api-openshift-ca\") pod \"332adcf3-f9d0-4c18-ab77-c358907288f3\" (UID: \"332adcf3-f9d0-4c18-ab77-c358907288f3\") " Nov 03 15:25:03.082091 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:03.082061 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/332adcf3-f9d0-4c18-ab77-c358907288f3-maas-api-openshift-ca" pod "332adcf3-f9d0-4c18-ab77-c358907288f3" (UID: "332adcf3-f9d0-4c18-ab77-c358907288f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:25:03.084095 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:03.084065 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332adcf3-f9d0-4c18-ab77-c358907288f3-kube-api-access-mc8sk" pod "332adcf3-f9d0-4c18-ab77-c358907288f3" (UID: "332adcf3-f9d0-4c18-ab77-c358907288f3"). InnerVolumeSpecName "kube-api-access-mc8sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:25:03.085854 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:03.085819 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/332adcf3-f9d0-4c18-ab77-c358907288f3-maas-api-xks-ca" pod "332adcf3-f9d0-4c18-ab77-c358907288f3" (UID: "332adcf3-f9d0-4c18-ab77-c358907288f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:25:03.183390 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:03.183356 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/332adcf3-f9d0-4c18-ab77-c358907288f3-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:03.183390 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:03.183384 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc8sk\" (UniqueName: \"kubernetes.io/projected/332adcf3-f9d0-4c18-ab77-c358907288f3-kube-api-access-mc8sk\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:03.183390 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:03.183395 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/332adcf3-f9d0-4c18-ab77-c358907288f3-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:03.791385 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:03.791357 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" Nov 03 15:25:03.791385 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:03.791380 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r" event={"ID":"332adcf3-f9d0-4c18-ab77-c358907288f3","Type":"ContainerDied","Data":"260eb04caf63249b19842bf8305cba6281dd76412cd6f3faba7adc51f53c993d"} Nov 03 15:25:03.791594 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:03.791425 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="260eb04caf63249b19842bf8305cba6281dd76412cd6f3faba7adc51f53c993d" Nov 03 15:25:08.704533 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:25:08.704440 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:25:14.563821 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.563790 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx"] Nov 03 15:25:14.564410 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.564388 4290 cpu_manager.go:414] "containerMap: removing container" podUID="332adcf3-f9d0-4c18-ab77-c358907288f3" containerName="revoke-keys" Nov 03 15:25:14.564410 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.564406 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332adcf3-f9d0-4c18-ab77-c358907288f3" containerName="revoke-keys" Nov 03 15:25:14.564550 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.564539 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="332adcf3-f9d0-4c18-ab77-c358907288f3" containerName="revoke-keys" Nov 03 15:25:14.568131 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.568112 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.621043 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.621011 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e62ff3-openshift-default-dockercfg-w2vlc\"" Nov 03 15:25:14.680617 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.680584 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx"] Nov 03 15:25:14.712450 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.712415 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.712637 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.712461 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12553958-5457-4c66-b011-431e954fef89-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.712637 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.712487 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12553958-5457-4c66-b011-431e954fef89-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.712637 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.712568 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8jmg\" (UniqueName: \"kubernetes.io/projected/12553958-5457-4c66-b011-431e954fef89-kube-api-access-j8jmg\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.712637 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.712602 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.712803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.712692 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12553958-5457-4c66-b011-431e954fef89-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.712803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.712771 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.712869 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.712823 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.712869 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.712844 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.813679 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.813647 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.813939 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.813763 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.813939 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.813797 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.813939 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.813849 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.813939 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.813923 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12553958-5457-4c66-b011-431e954fef89-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.814162 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.813976 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12553958-5457-4c66-b011-431e954fef89-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.814162 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.814012 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8jmg\" (UniqueName: \"kubernetes.io/projected/12553958-5457-4c66-b011-431e954fef89-kube-api-access-j8jmg\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.814162 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.814055 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.814162 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.814128 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12553958-5457-4c66-b011-431e954fef89-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.814162 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.814133 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.814162 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.814150 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.814465 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.814194 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.814465 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.814447 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.814800 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.814775 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12553958-5457-4c66-b011-431e954fef89-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.816463 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.816442 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.816558 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.816533 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12553958-5457-4c66-b011-431e954fef89-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.876051 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.876015 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12553958-5457-4c66-b011-431e954fef89-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.876220 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.876195 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8jmg\" (UniqueName: \"kubernetes.io/projected/12553958-5457-4c66-b011-431e954fef89-kube-api-access-j8jmg\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:14.882973 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:14.882950 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:15.322568 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:15.322540 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx"] Nov 03 15:25:15.322898 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:25:15.322878 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12553958_5457_4c66_b011_431e954fef89.slice/crio-bfe5db684b12bafe8e7edde6c8859cc94597e9ee7071627938b36928c87e8678 WatchSource:0}: Error finding container bfe5db684b12bafe8e7edde6c8859cc94597e9ee7071627938b36928c87e8678: Status 404 returned error can't find the container with id bfe5db684b12bafe8e7edde6c8859cc94597e9ee7071627938b36928c87e8678 Nov 03 15:25:15.325068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:15.325031 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:25:15.325155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:15.325098 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:25:15.325155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:15.325126 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:25:15.886305 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:15.886265 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" event={"ID":"12553958-5457-4c66-b011-431e954fef89","Type":"ContainerStarted","Data":"223b7fb759f6e329c0c7b408e1f0caa8a809146082363a517919c81d8ba1dade"} Nov 03 15:25:15.886952 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:15.886332 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" event={"ID":"12553958-5457-4c66-b011-431e954fef89","Type":"ContainerStarted","Data":"bfe5db684b12bafe8e7edde6c8859cc94597e9ee7071627938b36928c87e8678"} Nov 03 15:25:16.010469 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:16.010416 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" podStartSLOduration=2.010400951 podStartE2EDuration="2.010400951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:16.008079093 +0000 UTC m=+1023.439036899" watchObservedRunningTime="2026-11-03 15:25:16.010400951 +0000 UTC m=+1023.441358758" Nov 03 15:25:16.883719 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:16.883652 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:16.885309 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:16.885281 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 15:25:16.885798 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:16.885748 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" podUID="12553958-5457-4c66-b011-431e954fef89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 15:25:17.883887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:17.883853 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 15:25:17.884374 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:17.883914 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" podUID="12553958-5457-4c66-b011-431e954fef89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 15:25:18.883812 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:18.883781 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 15:25:18.884030 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:18.883840 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" podUID="12553958-5457-4c66-b011-431e954fef89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 15:25:19.883565 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:19.883531 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 15:25:19.883763 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:19.883599 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" podUID="12553958-5457-4c66-b011-431e954fef89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 15:25:20.884331 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:20.884295 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 15:25:20.884760 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:20.884351 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" podUID="12553958-5457-4c66-b011-431e954fef89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 15:25:21.883811 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:21.883774 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 15:25:21.883998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:21.883829 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" podUID="12553958-5457-4c66-b011-431e954fef89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 15:25:23.098000 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:23.097187 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:23.100892 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:23.100865 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:23.103176 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:23.103152 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:25:26.744486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.744388 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz"] Nov 03 15:25:26.751342 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.751315 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.812162 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.812131 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz"] Nov 03 15:25:26.867449 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.867415 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.867654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.867458 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8wp\" (UniqueName: \"kubernetes.io/projected/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-kube-api-access-vp8wp\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.867654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.867491 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.867654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.867535 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.867871 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.867763 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.867871 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.867810 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.867972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.867906 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.867972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.867936 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.868080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.868004 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.968863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.968826 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.969054 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.968893 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.969054 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.968919 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8wp\" (UniqueName: \"kubernetes.io/projected/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-kube-api-access-vp8wp\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.969054 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.968938 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.969054 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.968957 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.969054 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.969034 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.969357 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.969080 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.969357 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.969233 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.969357 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.969283 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.969755 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.969728 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.969867 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.969769 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.969867 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.969802 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.969990 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.969970 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.970212 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.970190 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.971994 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.971970 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:26.972253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:26.972233 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:27.031286 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:27.031211 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:27.031410 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:27.031286 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8wp\" (UniqueName: \"kubernetes.io/projected/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-kube-api-access-vp8wp\") pod \"maas-default-gateway-openshift-default-596dc9545f-nm4sz\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:27.069291 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:27.069256 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:27.514054 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:27.514016 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz"] Nov 03 15:25:27.514606 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:25:27.514573 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b80c4ee_4cf9_4cde_9e2f_1d4d72f0319e.slice/crio-a6219826464706f530fe67c18681fb36cad4828ab108430a70151e494654d542 WatchSource:0}: Error finding container a6219826464706f530fe67c18681fb36cad4828ab108430a70151e494654d542: Status 404 returned error can't find the container with id a6219826464706f530fe67c18681fb36cad4828ab108430a70151e494654d542 Nov 03 15:25:27.517175 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:27.517136 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:25:27.517281 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:27.517222 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:25:27.517281 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:27.517269 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:25:27.991399 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:27.991359 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" event={"ID":"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e","Type":"ContainerStarted","Data":"329b780fceb9d3e4217873ee3192ff75c2a45244adc17de72892b323f0f759ac"} Nov 03 15:25:27.991888 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:27.991422 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" event={"ID":"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e","Type":"ContainerStarted","Data":"a6219826464706f530fe67c18681fb36cad4828ab108430a70151e494654d542"} Nov 03 15:25:28.069518 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:28.069472 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:28.071157 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:28.071124 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nm4sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 15:25:28.071298 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:28.071179 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" podUID="3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 15:25:28.113224 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:28.113170 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" podStartSLOduration=2.113151884 podStartE2EDuration="2.113151884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:28.112314322 +0000 UTC m=+1035.543272152" watchObservedRunningTime="2026-11-03 15:25:28.113151884 +0000 UTC m=+1035.544109690" Nov 03 15:25:29.076223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:29.076181 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nm4sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 15:25:29.076736 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:29.076249 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" podUID="3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 15:25:30.070618 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:30.070584 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nm4sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 15:25:30.070852 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:30.070645 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" podUID="3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 15:25:31.070121 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:31.070088 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nm4sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 15:25:31.070605 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:31.070564 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" podUID="3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 15:25:32.070332 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:32.070291 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nm4sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 15:25:32.071042 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:32.071009 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" podUID="3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 15:25:33.070425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:33.070380 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nm4sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 15:25:33.070841 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:33.070440 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" podUID="3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 15:25:34.070754 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:34.070687 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nm4sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 15:25:34.071223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:34.070774 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" podUID="3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 15:25:35.070482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:35.070443 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nm4sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 15:25:35.070661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:35.070511 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" podUID="3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 15:25:36.329388 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.329357 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f"] Nov 03 15:25:36.333581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.333564 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.394360 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.394323 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f"] Nov 03 15:25:36.403964 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.403931 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed595ae7-8d05-42d9-84e8-4a370338dcb8-tls-certs\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.404184 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.404018 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-dshm\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.404184 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.404041 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq8mf\" (UniqueName: \"kubernetes.io/projected/ed595ae7-8d05-42d9-84e8-4a370338dcb8-kube-api-access-sq8mf\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.404184 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.404095 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-home\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.404184 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.404114 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-tmp-dir\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.404184 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.404163 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-model-cache\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.411427 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.411399 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:36.412281 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.412258 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:36.412916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.412900 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:25:36.437297 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.437236 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8b841a\"/\"openshift-service-ca.crt\"" Nov 03 15:25:36.437297 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.437236 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8b841a\"/\"default-dockercfg-sz8vc\"" Nov 03 15:25:36.437694 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.437669 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8b841a\"/\"kube-root-ca.crt\"" Nov 03 15:25:36.441912 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.441886 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8b841a\"/\"distinct-w3-8b841a-kserve-self-signed-certs\"" Nov 03 15:25:36.504931 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.504886 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed595ae7-8d05-42d9-84e8-4a370338dcb8-tls-certs\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.505111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.504997 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-dshm\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.505111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.505030 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq8mf\" (UniqueName: \"kubernetes.io/projected/ed595ae7-8d05-42d9-84e8-4a370338dcb8-kube-api-access-sq8mf\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.505262 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.505129 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-home\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.505262 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.505158 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-tmp-dir\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.505370 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.505261 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-model-cache\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.505622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.505600 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-tmp-dir\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.506084 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.506061 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-model-cache\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.506225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.506207 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-home\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.507807 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.507779 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed595ae7-8d05-42d9-84e8-4a370338dcb8-tls-certs\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.508057 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.508042 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-dshm\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.567922 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.567886 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq8mf\" (UniqueName: \"kubernetes.io/projected/ed595ae7-8d05-42d9-84e8-4a370338dcb8-kube-api-access-sq8mf\") pod \"distinct-w3-8b841a-kserve-7cddc66746-t6g4f\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:36.651458 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:36.651415 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:37.054828 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:37.054794 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f"] Nov 03 15:25:37.056271 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:25:37.056239 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded595ae7_8d05_42d9_84e8_4a370338dcb8.slice/crio-96397a6887cfe10725770b930940877d7229b267f4ed0cf0499b3ed6d409d310 WatchSource:0}: Error finding container 96397a6887cfe10725770b930940877d7229b267f4ed0cf0499b3ed6d409d310: Status 404 returned error can't find the container with id 96397a6887cfe10725770b930940877d7229b267f4ed0cf0499b3ed6d409d310 Nov 03 15:25:37.075582 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:37.073539 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" event={"ID":"ed595ae7-8d05-42d9-84e8-4a370338dcb8","Type":"ContainerStarted","Data":"96397a6887cfe10725770b930940877d7229b267f4ed0cf0499b3ed6d409d310"} Nov 03 15:25:38.086072 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:38.086024 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" event={"ID":"ed595ae7-8d05-42d9-84e8-4a370338dcb8","Type":"ContainerStarted","Data":"8705d5224f9a26868f51bd2dbabbf4cfb7e8fd8061e592dfb36338ae2345663e"} Nov 03 15:25:38.207741 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:38.207668 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podStartSLOduration=2.207652684 podStartE2EDuration="2.207652684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:38.204446676 +0000 UTC m=+1045.635404481" watchObservedRunningTime="2026-11-03 15:25:38.207652684 +0000 UTC m=+1045.638610497" Nov 03 15:25:38.652494 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:38.652461 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:38.703093 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:25:38.703059 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:25:41.104032 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.103990 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9"] Nov 03 15:25:41.108343 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.108323 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.167357 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.167325 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9"] Nov 03 15:25:41.212636 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.212598 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e62ff3\"/\"openshift-service-ca.crt\"" Nov 03 15:25:41.212636 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.212612 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e62ff3\"/\"default-dockercfg-gw6b2\"" Nov 03 15:25:41.213246 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.213229 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e62ff3\"/\"distinct-w0-e62ff3-kserve-self-signed-certs\"" Nov 03 15:25:41.213337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.213320 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e62ff3\"/\"kube-root-ca.crt\"" Nov 03 15:25:41.268269 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.268233 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-dshm\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.268269 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.268268 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrmt5\" (UniqueName: \"kubernetes.io/projected/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-kube-api-access-jrmt5\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.268531 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.268479 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-tls-certs\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.268531 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.268520 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-model-cache\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.268869 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.268580 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-tmp-dir\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.268869 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.268604 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-home\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.370143 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.370047 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-tls-certs\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.370143 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.370102 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-model-cache\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.370143 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.370129 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-tmp-dir\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.370474 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.370154 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-home\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.370474 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.370209 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-dshm\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.370474 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.370240 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrmt5\" (UniqueName: \"kubernetes.io/projected/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-kube-api-access-jrmt5\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.370621 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.370569 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-model-cache\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.370817 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.370773 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-home\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.370918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.370832 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-tmp-dir\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.373548 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.373508 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-tls-certs\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.373948 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.373920 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-dshm\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.432506 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.432470 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrmt5\" (UniqueName: \"kubernetes.io/projected/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-kube-api-access-jrmt5\") pod \"distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:41.722334 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:41.722300 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:42.111176 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:25:42.111149 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6203a8f_8dc0_4b4b_8d1f_eb6adb071ae3.slice/crio-f263bd25f117b48392ea9595482394fcd398570bcd9989c91d7fdde9715b3f02 WatchSource:0}: Error finding container f263bd25f117b48392ea9595482394fcd398570bcd9989c91d7fdde9715b3f02: Status 404 returned error can't find the container with id f263bd25f117b48392ea9595482394fcd398570bcd9989c91d7fdde9715b3f02 Nov 03 15:25:42.111524 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:42.111259 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9"] Nov 03 15:25:43.124419 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:43.124372 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" event={"ID":"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3","Type":"ContainerStarted","Data":"f06fb4bbb4d8f9d29cf19ee825fccf477f23c22a620d0c8f4fa3cdcc582a8abc"} Nov 03 15:25:43.124968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:43.124431 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" event={"ID":"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3","Type":"ContainerStarted","Data":"f263bd25f117b48392ea9595482394fcd398570bcd9989c91d7fdde9715b3f02"} Nov 03 15:25:43.283191 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:43.283124 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podStartSLOduration=2.283109175 podStartE2EDuration="2.283109175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:43.281226182 +0000 UTC m=+1050.712183989" watchObservedRunningTime="2026-11-03 15:25:43.283109175 +0000 UTC m=+1050.714066981" Nov 03 15:25:43.722565 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:43.722536 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:44.130018 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.129979 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct"] Nov 03 15:25:44.134305 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.134281 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.188285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.188250 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-bc1f07-openshift-default-dockercfg-lrpxt\"" Nov 03 15:25:44.241866 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.241832 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct"] Nov 03 15:25:44.304601 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.304561 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.304790 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.304665 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.304790 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.304751 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz7s6\" (UniqueName: \"kubernetes.io/projected/c5a211f3-7585-4727-bb04-b468c5c6e89f-kube-api-access-pz7s6\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.304953 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.304836 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.305090 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.305065 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a211f3-7585-4727-bb04-b468c5c6e89f-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.305210 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.305191 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.305273 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.305227 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.305367 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.305345 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.305398 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.305383 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.406806 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.406695 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.406806 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.406752 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.406806 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.406792 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.406806 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.406808 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.407146 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.406851 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.407146 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.406888 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.407146 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.406948 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz7s6\" (UniqueName: \"kubernetes.io/projected/c5a211f3-7585-4727-bb04-b468c5c6e89f-kube-api-access-pz7s6\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.407146 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.407011 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.407146 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.407121 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a211f3-7585-4727-bb04-b468c5c6e89f-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.407393 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.407173 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.407393 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.407271 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.407494 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.407412 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.407494 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.407443 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.407870 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.407851 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a211f3-7585-4727-bb04-b468c5c6e89f-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.409647 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.409629 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.409737 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.409683 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.469558 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.469519 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.470097 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.470073 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz7s6\" (UniqueName: \"kubernetes.io/projected/c5a211f3-7585-4727-bb04-b468c5c6e89f-kube-api-access-pz7s6\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:44.750306 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:44.750223 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:45.199208 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:45.199172 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct"] Nov 03 15:25:45.234776 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:25:45.234742 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a211f3_7585_4727_bb04_b468c5c6e89f.slice/crio-610afe3f82918349d4d76b4e81d50c2b357cae4dce682a2020edd3510f821854 WatchSource:0}: Error finding container 610afe3f82918349d4d76b4e81d50c2b357cae4dce682a2020edd3510f821854: Status 404 returned error can't find the container with id 610afe3f82918349d4d76b4e81d50c2b357cae4dce682a2020edd3510f821854 Nov 03 15:25:45.237200 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:45.237165 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:25:45.237280 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:45.237237 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:25:45.237280 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:45.237268 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:25:46.150323 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:46.150285 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" event={"ID":"c5a211f3-7585-4727-bb04-b468c5c6e89f","Type":"ContainerStarted","Data":"3cf928d71649044f9cb06c332e9262390bed0094bfb9934a9a78624d88f36fb3"} Nov 03 15:25:46.150514 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:46.150350 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" event={"ID":"c5a211f3-7585-4727-bb04-b468c5c6e89f","Type":"ContainerStarted","Data":"610afe3f82918349d4d76b4e81d50c2b357cae4dce682a2020edd3510f821854"} Nov 03 15:25:46.325839 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:46.325782 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" podStartSLOduration=2.325768076 podStartE2EDuration="2.325768076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:46.322118741 +0000 UTC m=+1053.753076571" watchObservedRunningTime="2026-11-03 15:25:46.325768076 +0000 UTC m=+1053.756725882" Nov 03 15:25:46.667894 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:46.666937 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:46.687610 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:46.687586 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:46.757726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:46.756870 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:46.771759 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:46.771681 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 15:25:46.771921 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:46.771797 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" podUID="c5a211f3-7585-4727-bb04-b468c5c6e89f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 15:25:47.194784 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:47.194756 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:25:47.750929 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:47.750886 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 15:25:47.751624 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:47.751574 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" podUID="c5a211f3-7585-4727-bb04-b468c5c6e89f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 15:25:48.751649 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:48.751162 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 15:25:48.751649 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:48.751265 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" podUID="c5a211f3-7585-4727-bb04-b468c5c6e89f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 15:25:49.750956 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:49.750920 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 15:25:49.751256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:49.751226 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" podUID="c5a211f3-7585-4727-bb04-b468c5c6e89f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 15:25:50.756165 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:50.756125 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 15:25:50.756801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:50.756767 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" podUID="c5a211f3-7585-4727-bb04-b468c5c6e89f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 15:25:51.722664 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:51.722623 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:51.756977 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:51.756940 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 15:25:51.757509 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:51.757393 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" podUID="c5a211f3-7585-4727-bb04-b468c5c6e89f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 15:25:51.757774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:51.757749 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:52.244631 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:52.244597 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:25:53.298486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:53.298451 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:53.299247 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:53.299224 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:25:53.300315 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:25:53.300295 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:26:08.709193 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:26:08.709107 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:26:26.043103 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.043068 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n"] Nov 03 15:26:26.049066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.049040 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.105471 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.105438 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n"] Nov 03 15:26:26.150987 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.150879 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bc1f07\"/\"distinct-w4-bc1f07-kserve-self-signed-certs\"" Nov 03 15:26:26.150987 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.151000 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bc1f07\"/\"default-dockercfg-44z2w\"" Nov 03 15:26:26.151504 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.151485 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bc1f07\"/\"openshift-service-ca.crt\"" Nov 03 15:26:26.151609 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.151517 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bc1f07\"/\"kube-root-ca.crt\"" Nov 03 15:26:26.158795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.158770 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k7r2\" (UniqueName: \"kubernetes.io/projected/17b58cf9-2cff-4648-86c1-0b1d09acacdd-kube-api-access-9k7r2\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.158904 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.158838 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-home\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.158953 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.158935 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b58cf9-2cff-4648-86c1-0b1d09acacdd-tls-certs\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.159004 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.158994 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-tmp-dir\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.159052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.159022 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-dshm\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.159052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.159045 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-model-cache\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.260619 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.260569 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b58cf9-2cff-4648-86c1-0b1d09acacdd-tls-certs\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.260843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.260649 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-tmp-dir\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.260843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.260692 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-dshm\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.260843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.260767 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-model-cache\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.261034 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.260885 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k7r2\" (UniqueName: \"kubernetes.io/projected/17b58cf9-2cff-4648-86c1-0b1d09acacdd-kube-api-access-9k7r2\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.261034 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.260973 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-home\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.261241 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.261217 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-tmp-dir\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.261534 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.261507 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-model-cache\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.261658 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.261551 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-home\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.263532 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.263510 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-dshm\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.263653 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.263559 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b58cf9-2cff-4648-86c1-0b1d09acacdd-tls-certs\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.325155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.325071 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k7r2\" (UniqueName: \"kubernetes.io/projected/17b58cf9-2cff-4648-86c1-0b1d09acacdd-kube-api-access-9k7r2\") pod \"distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.362972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.362936 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:26.772821 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:26.772792 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n"] Nov 03 15:26:26.774413 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:26:26.774384 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17b58cf9_2cff_4648_86c1_0b1d09acacdd.slice/crio-1c6509ff6cb96b1092590708238aa973232a9a99848e4fb609a5a6be94ce63ed WatchSource:0}: Error finding container 1c6509ff6cb96b1092590708238aa973232a9a99848e4fb609a5a6be94ce63ed: Status 404 returned error can't find the container with id 1c6509ff6cb96b1092590708238aa973232a9a99848e4fb609a5a6be94ce63ed Nov 03 15:26:27.509189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:27.509152 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" event={"ID":"17b58cf9-2cff-4648-86c1-0b1d09acacdd","Type":"ContainerStarted","Data":"a15ed5bab533a29f8fc286a0cbe6b43a442ec5ba2e99342e72054484cd97a793"} Nov 03 15:26:27.509783 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:27.509202 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" event={"ID":"17b58cf9-2cff-4648-86c1-0b1d09acacdd","Type":"ContainerStarted","Data":"1c6509ff6cb96b1092590708238aa973232a9a99848e4fb609a5a6be94ce63ed"} Nov 03 15:26:27.632281 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:27.632227 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podStartSLOduration=2.632213128 podStartE2EDuration="2.632213128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:26:27.628960654 +0000 UTC m=+1095.059918461" watchObservedRunningTime="2026-11-03 15:26:27.632213128 +0000 UTC m=+1095.063170934" Nov 03 15:26:28.363387 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:28.363357 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:36.363764 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:36.363727 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:36.382397 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:36.382372 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:36.590428 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:36.590401 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:26:38.713045 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:26:38.712940 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:26:51.402964 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.402920 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv"] Nov 03 15:26:51.407666 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.407642 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.511434 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.511405 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-34e8c5\"/\"default-dockercfg-sqn5c\"" Nov 03 15:26:51.511663 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.511406 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-34e8c5\"/\"kube-root-ca.crt\"" Nov 03 15:26:51.511663 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.511406 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-34e8c5\"/\"openshift-service-ca.crt\"" Nov 03 15:26:51.511663 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.511504 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-34e8c5\"/\"distinct-w5-34e8c5-kserve-self-signed-certs\"" Nov 03 15:26:51.515550 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.515527 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv"] Nov 03 15:26:51.568232 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.568184 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm7m6\" (UniqueName: \"kubernetes.io/projected/2de980d4-51ff-4fad-a85a-2f45af90a354-kube-api-access-gm7m6\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.568431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.568288 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-tmp-dir\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.568431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.568334 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-home\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.568431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.568368 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2de980d4-51ff-4fad-a85a-2f45af90a354-tls-certs\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.568609 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.568473 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-model-cache\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.568609 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.568570 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-dshm\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.669815 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.669718 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-model-cache\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.669994 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.669821 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-dshm\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.669994 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.669891 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm7m6\" (UniqueName: \"kubernetes.io/projected/2de980d4-51ff-4fad-a85a-2f45af90a354-kube-api-access-gm7m6\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.669994 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.669961 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-tmp-dir\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.670159 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.670002 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-home\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.670159 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.670032 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2de980d4-51ff-4fad-a85a-2f45af90a354-tls-certs\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.670159 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.670099 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-model-cache\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.670321 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.670278 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-tmp-dir\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.670373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.670354 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-home\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.672267 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.672246 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-dshm\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.672667 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.672648 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2de980d4-51ff-4fad-a85a-2f45af90a354-tls-certs\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:51.732562 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:51.732522 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm7m6\" (UniqueName: \"kubernetes.io/projected/2de980d4-51ff-4fad-a85a-2f45af90a354-kube-api-access-gm7m6\") pod \"distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:52.021236 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:52.021151 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:26:52.405411 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:52.405377 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv"] Nov 03 15:26:52.406037 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:26:52.406008 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2de980d4_51ff_4fad_a85a_2f45af90a354.slice/crio-b09da9f3769f66793b32ccfe5d921e71af44392715e24d3968401e21f89ef1bd WatchSource:0}: Error finding container b09da9f3769f66793b32ccfe5d921e71af44392715e24d3968401e21f89ef1bd: Status 404 returned error can't find the container with id b09da9f3769f66793b32ccfe5d921e71af44392715e24d3968401e21f89ef1bd Nov 03 15:26:52.703087 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:52.702998 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" event={"ID":"2de980d4-51ff-4fad-a85a-2f45af90a354","Type":"ContainerStarted","Data":"7e0787a54e7ca4982f707483fa104e8255d9795e1085d65b135176d793197269"} Nov 03 15:26:52.703087 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:52.703059 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" event={"ID":"2de980d4-51ff-4fad-a85a-2f45af90a354","Type":"ContainerStarted","Data":"b09da9f3769f66793b32ccfe5d921e71af44392715e24d3968401e21f89ef1bd"} Nov 03 15:26:52.820983 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:52.820932 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podStartSLOduration=1.820914099 podStartE2EDuration="1.820914099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:26:52.82051287 +0000 UTC m=+1120.251470676" watchObservedRunningTime="2026-11-03 15:26:52.820914099 +0000 UTC m=+1120.251871935" Nov 03 15:26:53.022153 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:26:53.022063 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:27:02.022406 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:02.022329 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:27:02.035182 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:02.035159 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:27:02.792127 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:02.792091 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:27:08.707219 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:27:08.707130 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:27:15.825323 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:15.825286 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j"] Nov 03 15:27:15.829764 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:15.829746 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:15.883128 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:15.883095 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f78fd6-openshift-default-dockercfg-cd4bc\"" Nov 03 15:27:15.936827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:15.936796 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j"] Nov 03 15:27:15.984403 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:15.984368 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:15.984601 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:15.984435 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85875\" (UniqueName: \"kubernetes.io/projected/fa7c2436-de33-42f9-b8c9-db8fc7d48229-kube-api-access-85875\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:15.984601 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:15.984541 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:15.984601 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:15.984594 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:15.984773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:15.984654 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:15.984773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:15.984689 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:15.984894 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:15.984812 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:15.984947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:15.984896 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:15.985007 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:15.984992 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.086417 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.086335 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85875\" (UniqueName: \"kubernetes.io/projected/fa7c2436-de33-42f9-b8c9-db8fc7d48229-kube-api-access-85875\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.086417 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.086380 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.086780 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.086526 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.086780 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.086556 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.086780 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.086579 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.086780 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.086609 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.086780 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.086682 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.087073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.086788 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.087073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.086844 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.087073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.086912 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.087073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.086957 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.087355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.087148 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.087425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.087367 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.087575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.087556 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.089499 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.089476 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.089712 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.089673 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.148394 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.148366 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.148737 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.148684 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85875\" (UniqueName: \"kubernetes.io/projected/fa7c2436-de33-42f9-b8c9-db8fc7d48229-kube-api-access-85875\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.444262 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.444222 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:16.888741 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:27:16.888691 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7c2436_de33_42f9_b8c9_db8fc7d48229.slice/crio-f60dada3c8c24074f52529d7897fd0522b02f16a27de15bd114687290183aa92 WatchSource:0}: Error finding container f60dada3c8c24074f52529d7897fd0522b02f16a27de15bd114687290183aa92: Status 404 returned error can't find the container with id f60dada3c8c24074f52529d7897fd0522b02f16a27de15bd114687290183aa92 Nov 03 15:27:16.889518 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.889475 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j"] Nov 03 15:27:16.891350 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.891319 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:27:16.891422 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.891387 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:27:16.891457 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.891420 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:27:16.901345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:16.901317 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" event={"ID":"fa7c2436-de33-42f9-b8c9-db8fc7d48229","Type":"ContainerStarted","Data":"f60dada3c8c24074f52529d7897fd0522b02f16a27de15bd114687290183aa92"} Nov 03 15:27:17.910425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:17.910389 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" event={"ID":"fa7c2436-de33-42f9-b8c9-db8fc7d48229","Type":"ContainerStarted","Data":"bafcb72f3f88876a193cc4b0ab208c5b6393f610841c4d5506a9f31e40d2e3fd"} Nov 03 15:27:18.172171 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:18.172039 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" podStartSLOduration=3.172017843 podStartE2EDuration="3.172017843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:27:18.09192499 +0000 UTC m=+1145.522882805" watchObservedRunningTime="2026-11-03 15:27:18.172017843 +0000 UTC m=+1145.602975653" Nov 03 15:27:18.456786 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:18.456068 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:18.461726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:18.459625 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 15:27:18.461726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:18.459689 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" podUID="fa7c2436-de33-42f9-b8c9-db8fc7d48229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 15:27:19.445737 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:19.445150 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 15:27:19.445737 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:19.445226 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" podUID="fa7c2436-de33-42f9-b8c9-db8fc7d48229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 15:27:20.456217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:20.456178 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 15:27:20.456871 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:20.456836 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" podUID="fa7c2436-de33-42f9-b8c9-db8fc7d48229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 15:27:21.456866 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:21.456826 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 15:27:21.457957 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:21.457923 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" podUID="fa7c2436-de33-42f9-b8c9-db8fc7d48229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 15:27:22.453164 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:22.453122 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 15:27:22.454212 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:22.454179 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" podUID="fa7c2436-de33-42f9-b8c9-db8fc7d48229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 15:27:23.445305 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:23.445264 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 15:27:23.445860 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:23.445341 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" podUID="fa7c2436-de33-42f9-b8c9-db8fc7d48229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 15:27:25.043441 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:25.042071 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:25.043441 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:25.043289 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:25.053294 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:25.053250 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:27:38.705134 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:27:38.705100 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:27:58.719846 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.719812 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl"] Nov 03 15:27:58.723151 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.723134 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.781351 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.781324 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f59d26\"/\"distinct-2-w6-f59d26-kserve-self-signed-certs\"" Nov 03 15:27:58.815337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.815300 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-home\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.815337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.815339 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-tmp-dir\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.815621 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.815369 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-dshm\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.815621 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.815402 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b83d9a7-0109-4660-aa46-a64c544fcb96-tls-certs\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.815744 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.815638 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgjcw\" (UniqueName: \"kubernetes.io/projected/4b83d9a7-0109-4660-aa46-a64c544fcb96-kube-api-access-hgjcw\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.815744 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.815669 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-model-cache\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.835550 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.835519 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl"] Nov 03 15:27:58.917301 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.917264 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b83d9a7-0109-4660-aa46-a64c544fcb96-tls-certs\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.917849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.917366 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgjcw\" (UniqueName: \"kubernetes.io/projected/4b83d9a7-0109-4660-aa46-a64c544fcb96-kube-api-access-hgjcw\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.917849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.917472 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-model-cache\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.917849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.917627 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-home\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.917849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.917655 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-tmp-dir\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.917849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.917756 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-dshm\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.918280 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.917914 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-model-cache\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.918280 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.917937 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-home\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.918280 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.918026 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-tmp-dir\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.920003 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.919980 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-dshm\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.920209 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.920187 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b83d9a7-0109-4660-aa46-a64c544fcb96-tls-certs\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:58.988135 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:58.988056 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgjcw\" (UniqueName: \"kubernetes.io/projected/4b83d9a7-0109-4660-aa46-a64c544fcb96-kube-api-access-hgjcw\") pod \"distinct-2-w6-f59d26-kserve-d57dbb576-thbvl\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:59.036206 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:59.036165 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:27:59.421427 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:27:59.421395 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl"] Nov 03 15:27:59.422053 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:27:59.422027 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b83d9a7_0109_4660_aa46_a64c544fcb96.slice/crio-1ca38ce4ba1012031156a975699792d4ab9d6e311e8432eaa8ded69f6e7ea1fd WatchSource:0}: Error finding container 1ca38ce4ba1012031156a975699792d4ab9d6e311e8432eaa8ded69f6e7ea1fd: Status 404 returned error can't find the container with id 1ca38ce4ba1012031156a975699792d4ab9d6e311e8432eaa8ded69f6e7ea1fd Nov 03 15:28:00.326781 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:00.326737 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" event={"ID":"4b83d9a7-0109-4660-aa46-a64c544fcb96","Type":"ContainerStarted","Data":"251d68f04ec0c8eca621a49a0232d46d7a29cca8feff72472b7e9e30e823d98a"} Nov 03 15:28:00.327192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:00.326798 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" event={"ID":"4b83d9a7-0109-4660-aa46-a64c544fcb96","Type":"ContainerStarted","Data":"1ca38ce4ba1012031156a975699792d4ab9d6e311e8432eaa8ded69f6e7ea1fd"} Nov 03 15:28:00.450412 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:00.450345 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podStartSLOduration=2.450325057 podStartE2EDuration="2.450325057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:28:00.444025747 +0000 UTC m=+1187.874983553" watchObservedRunningTime="2026-11-03 15:28:00.450325057 +0000 UTC m=+1187.881282865" Nov 03 15:28:01.036417 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:01.036378 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:28:08.709460 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:28:08.709367 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:28:09.036874 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:09.036781 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:28:09.054477 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:09.054438 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:28:09.416431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:09.416401 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:28:12.848570 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:12.848542 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:28:12.849642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:12.849621 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:28:12.851376 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:12.851358 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:28:12.852372 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:12.852354 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:28:16.012611 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.012572 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk"] Nov 03 15:28:16.017133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.017115 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.101834 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.101798 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk"] Nov 03 15:28:16.141004 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.140959 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlrl\" (UniqueName: \"kubernetes.io/projected/2af12a1b-e8c5-418d-b61d-f9928a8213b6-kube-api-access-qmlrl\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.141211 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.141114 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.141211 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.141159 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.141331 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.141241 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.141331 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.141311 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.141436 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.141364 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.141490 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.141449 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.141490 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.141475 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.141599 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.141502 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.242418 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.242382 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.242626 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.242454 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlrl\" (UniqueName: \"kubernetes.io/projected/2af12a1b-e8c5-418d-b61d-f9928a8213b6-kube-api-access-qmlrl\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.242626 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.242549 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.242626 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.242574 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.242976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.242631 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.242976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.242692 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.242976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.242766 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.242976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.242835 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.242976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.242864 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.243353 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.243330 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.244066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.244041 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.246471 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.244886 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.246752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.246684 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.246900 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.246849 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.247012 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.246986 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.247155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.247131 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.305354 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.305275 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.305354 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.305307 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlrl\" (UniqueName: \"kubernetes.io/projected/2af12a1b-e8c5-418d-b61d-f9928a8213b6-kube-api-access-qmlrl\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.331664 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.331634 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:16.780424 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.780394 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk"] Nov 03 15:28:16.782508 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:28:16.782477 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af12a1b_e8c5_418d_b61d_f9928a8213b6.slice/crio-c5d99da0d926134bde278cee7bc8e1ab2c4500fbae19c8a47efec19e244f5926 WatchSource:0}: Error finding container c5d99da0d926134bde278cee7bc8e1ab2c4500fbae19c8a47efec19e244f5926: Status 404 returned error can't find the container with id c5d99da0d926134bde278cee7bc8e1ab2c4500fbae19c8a47efec19e244f5926 Nov 03 15:28:16.785307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.785272 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:28:16.785541 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.785512 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:28:16.785631 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:16.785577 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:28:17.475968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:17.475916 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" event={"ID":"2af12a1b-e8c5-418d-b61d-f9928a8213b6","Type":"ContainerStarted","Data":"0dbabd2c8da12d7dfbb44b4b22b2eea6ac61051d976d17c0c2eb55a87ec513fe"} Nov 03 15:28:17.476407 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:17.475994 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" event={"ID":"2af12a1b-e8c5-418d-b61d-f9928a8213b6","Type":"ContainerStarted","Data":"c5d99da0d926134bde278cee7bc8e1ab2c4500fbae19c8a47efec19e244f5926"} Nov 03 15:28:17.699285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:17.699228 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" podStartSLOduration=2.699212762 podStartE2EDuration="2.699212762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:28:17.647002752 +0000 UTC m=+1205.077960558" watchObservedRunningTime="2026-11-03 15:28:17.699212762 +0000 UTC m=+1205.130170567" Nov 03 15:28:18.332573 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:18.332533 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:18.334194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:18.334163 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 15:28:18.334356 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:18.334224 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" podUID="2af12a1b-e8c5-418d-b61d-f9928a8213b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 15:28:19.332424 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:19.332389 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 15:28:19.332835 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:19.332440 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" podUID="2af12a1b-e8c5-418d-b61d-f9928a8213b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 15:28:20.332788 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:20.332750 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 15:28:20.333195 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:20.332809 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" podUID="2af12a1b-e8c5-418d-b61d-f9928a8213b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 15:28:21.332607 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:21.332564 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 15:28:21.332828 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:21.332629 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" podUID="2af12a1b-e8c5-418d-b61d-f9928a8213b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 15:28:22.332671 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:22.332633 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 15:28:22.332877 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:22.332721 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" podUID="2af12a1b-e8c5-418d-b61d-f9928a8213b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 15:28:23.332959 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:23.332921 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 15:28:23.333458 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:23.332986 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" podUID="2af12a1b-e8c5-418d-b61d-f9928a8213b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 15:28:24.505406 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:24.505369 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:24.506640 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:24.506614 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:24.507569 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:24.507544 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:28:32.144618 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.144535 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g"] Nov 03 15:28:32.149645 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.149622 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.194857 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.194790 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.194857 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.194844 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.195179 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.194929 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f66f4-3203-4f72-bce5-c94005f38c35-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.195179 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.194990 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.195179 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.195038 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.195179 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.195145 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.195409 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.195242 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.195409 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.195349 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.195546 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.195454 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hf7w\" (UniqueName: \"kubernetes.io/projected/0a9f66f4-3203-4f72-bce5-c94005f38c35-kube-api-access-8hf7w\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.204390 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.204364 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f59d26-openshift-default-dockercfg-nkdw5\"" Nov 03 15:28:32.261204 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.261167 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g"] Nov 03 15:28:32.297126 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.297089 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.297321 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.297205 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.297321 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.297268 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.297321 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.297307 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.297488 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.297389 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hf7w\" (UniqueName: \"kubernetes.io/projected/0a9f66f4-3203-4f72-bce5-c94005f38c35-kube-api-access-8hf7w\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.297488 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.297465 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.297592 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.297509 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.297592 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.297547 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f66f4-3203-4f72-bce5-c94005f38c35-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.297592 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.297585 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.297845 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.297815 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.298183 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.298039 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.298293 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.298266 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.298534 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.298505 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f66f4-3203-4f72-bce5-c94005f38c35-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.298648 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.298625 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.299911 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.299877 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.300179 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.300162 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.359917 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.359888 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.360105 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.360085 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hf7w\" (UniqueName: \"kubernetes.io/projected/0a9f66f4-3203-4f72-bce5-c94005f38c35-kube-api-access-8hf7w\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.464176 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.464085 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:32.914397 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:28:32.914367 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9f66f4_3203_4f72_bce5_c94005f38c35.slice/crio-87138b4558f80a81cce4d96ff00cfed7d233943e02fc5b0945777e34ec30bf2a WatchSource:0}: Error finding container 87138b4558f80a81cce4d96ff00cfed7d233943e02fc5b0945777e34ec30bf2a: Status 404 returned error can't find the container with id 87138b4558f80a81cce4d96ff00cfed7d233943e02fc5b0945777e34ec30bf2a Nov 03 15:28:32.915589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.915565 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g"] Nov 03 15:28:32.916883 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.916849 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:28:32.916964 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.916910 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:28:32.916964 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:32.916941 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:28:33.616299 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:33.616264 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" event={"ID":"0a9f66f4-3203-4f72-bce5-c94005f38c35","Type":"ContainerStarted","Data":"16424b11aec3eec54c63a4117be2d547a01cdc6d4a18fe0d79eb14831674354b"} Nov 03 15:28:33.616734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:33.616329 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" event={"ID":"0a9f66f4-3203-4f72-bce5-c94005f38c35","Type":"ContainerStarted","Data":"87138b4558f80a81cce4d96ff00cfed7d233943e02fc5b0945777e34ec30bf2a"} Nov 03 15:28:33.856725 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:33.840785 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" podStartSLOduration=1.840763764 podStartE2EDuration="1.840763764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:28:33.788283725 +0000 UTC m=+1221.219241535" watchObservedRunningTime="2026-11-03 15:28:33.840763764 +0000 UTC m=+1221.271721570" Nov 03 15:28:34.465325 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:34.465276 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:34.466602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:34.466574 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:28:34.466838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:34.466809 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" podUID="0a9f66f4-3203-4f72-bce5-c94005f38c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 15:28:35.465137 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:35.465099 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:28:35.466539 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:35.466505 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" podUID="0a9f66f4-3203-4f72-bce5-c94005f38c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 15:28:36.470203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:36.470113 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:28:36.470203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:36.470177 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" podUID="0a9f66f4-3203-4f72-bce5-c94005f38c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 15:28:37.471741 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:37.470951 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:28:37.471741 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:37.471023 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" podUID="0a9f66f4-3203-4f72-bce5-c94005f38c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 15:28:38.473517 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:38.472036 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:28:38.473517 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:38.472109 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" podUID="0a9f66f4-3203-4f72-bce5-c94005f38c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 15:28:38.706645 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:28:38.706609 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:28:39.464530 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:39.464475 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:28:39.464809 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:39.464563 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" podUID="0a9f66f4-3203-4f72-bce5-c94005f38c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 15:28:40.481785 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:40.481756 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:40.483053 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:40.483028 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:28:40.483975 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:28:40.483952 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:29:00.999198 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:00.999163 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb"] Nov 03 15:29:01.005003 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.004966 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.061162 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.061129 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9855ff-openshift-default-dockercfg-5hjbk\"" Nov 03 15:29:01.098976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.098938 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.098976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.098974 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.099185 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.099119 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.099185 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.099146 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.099185 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.099172 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.099317 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.099302 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.099354 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.099327 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.099391 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.099363 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.099509 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.099495 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd77q\" (UniqueName: \"kubernetes.io/projected/6f7d95e8-3d59-481b-bdfd-e33e6854d805-kube-api-access-fd77q\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.116000 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.115965 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb"] Nov 03 15:29:01.200585 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.200535 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.200788 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.200636 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd77q\" (UniqueName: \"kubernetes.io/projected/6f7d95e8-3d59-481b-bdfd-e33e6854d805-kube-api-access-fd77q\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.200788 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.200662 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.200788 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.200679 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.201066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.200797 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.201066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.200822 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.201066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.200845 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.201066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.200953 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.201066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.201057 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.201337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.201079 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.201337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.201161 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.201457 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.201383 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.201515 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.201465 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.201580 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.201556 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.203357 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.203327 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.203601 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.203582 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.264375 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.264305 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.264513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.264438 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd77q\" (UniqueName: \"kubernetes.io/projected/6f7d95e8-3d59-481b-bdfd-e33e6854d805-kube-api-access-fd77q\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.320953 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.320915 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:01.770189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.770064 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb"] Nov 03 15:29:01.773188 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:29:01.773159 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7d95e8_3d59_481b_bdfd_e33e6854d805.slice/crio-69882af52ff8edf4a6d159da2eef8ba5088ffb7f1c0ef19d15058b2fc563b1d2 WatchSource:0}: Error finding container 69882af52ff8edf4a6d159da2eef8ba5088ffb7f1c0ef19d15058b2fc563b1d2: Status 404 returned error can't find the container with id 69882af52ff8edf4a6d159da2eef8ba5088ffb7f1c0ef19d15058b2fc563b1d2 Nov 03 15:29:01.775872 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.775840 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:29:01.775948 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.775909 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:29:01.775948 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.775940 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:29:01.870017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.869984 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" event={"ID":"6f7d95e8-3d59-481b-bdfd-e33e6854d805","Type":"ContainerStarted","Data":"864e8c064cb273491dfa65c3742afd06ae4ff405777a47e186c76d6fe6d02907"} Nov 03 15:29:01.870190 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.870051 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" event={"ID":"6f7d95e8-3d59-481b-bdfd-e33e6854d805","Type":"ContainerStarted","Data":"69882af52ff8edf4a6d159da2eef8ba5088ffb7f1c0ef19d15058b2fc563b1d2"} Nov 03 15:29:01.976269 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.976231 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g"] Nov 03 15:29:01.981681 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.981654 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:01.994168 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:01.994111 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" podStartSLOduration=1.994093178 podStartE2EDuration="1.994093178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:01.993514554 +0000 UTC m=+1249.424472360" watchObservedRunningTime="2026-11-03 15:29:01.994093178 +0000 UTC m=+1249.425050986" Nov 03 15:29:02.012249 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.012218 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.012669 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.012267 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.012669 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.012370 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.012669 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.012399 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a896388-6490-47c0-9eed-5c81fde73165-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.012669 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.012415 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7dkh\" (UniqueName: \"kubernetes.io/projected/5a896388-6490-47c0-9eed-5c81fde73165-kube-api-access-f7dkh\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.012669 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.012546 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a896388-6490-47c0-9eed-5c81fde73165-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.012669 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.012602 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.012669 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.012659 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a896388-6490-47c0-9eed-5c81fde73165-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.013001 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.012775 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.035496 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.034800 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8b841a-openshift-default-dockercfg-bszcv\"" Nov 03 15:29:02.054755 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.041721 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g"] Nov 03 15:29:02.114013 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.113973 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a896388-6490-47c0-9eed-5c81fde73165-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.114209 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.114075 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.114209 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.114099 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a896388-6490-47c0-9eed-5c81fde73165-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.114209 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.114164 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.114381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.114242 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.114381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.114306 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.114486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.114386 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.114486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.114418 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a896388-6490-47c0-9eed-5c81fde73165-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.114486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.114444 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7dkh\" (UniqueName: \"kubernetes.io/projected/5a896388-6490-47c0-9eed-5c81fde73165-kube-api-access-f7dkh\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.114762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.114683 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.114861 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.114777 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.114861 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.114813 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.115056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.115040 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.115372 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.115272 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a896388-6490-47c0-9eed-5c81fde73165-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.116839 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.116818 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a896388-6490-47c0-9eed-5c81fde73165-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.116903 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.116857 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.175816 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.175780 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a896388-6490-47c0-9eed-5c81fde73165-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.176000 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.175917 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7dkh\" (UniqueName: \"kubernetes.io/projected/5a896388-6490-47c0-9eed-5c81fde73165-kube-api-access-f7dkh\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.298340 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.298256 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:02.321474 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.321448 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:02.323576 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.323542 4290 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:02.323742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.323598 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" podUID="6f7d95e8-3d59-481b-bdfd-e33e6854d805" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 15:29:02.779629 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:29:02.779587 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a896388_6490_47c0_9eed_5c81fde73165.slice/crio-54db47ba24b18f13cc8f2729abb0c5364bda736a279c1ef9547f42f5b50fa382 WatchSource:0}: Error finding container 54db47ba24b18f13cc8f2729abb0c5364bda736a279c1ef9547f42f5b50fa382: Status 404 returned error can't find the container with id 54db47ba24b18f13cc8f2729abb0c5364bda736a279c1ef9547f42f5b50fa382 Nov 03 15:29:02.780560 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.780536 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g"] Nov 03 15:29:02.783180 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.783139 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:29:02.783242 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.783227 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:29:02.783286 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.783270 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:29:02.887266 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:02.887235 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" event={"ID":"5a896388-6490-47c0-9eed-5c81fde73165","Type":"ContainerStarted","Data":"54db47ba24b18f13cc8f2729abb0c5364bda736a279c1ef9547f42f5b50fa382"} Nov 03 15:29:03.321661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:03.321629 4290 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:03.322154 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:03.321686 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" podUID="6f7d95e8-3d59-481b-bdfd-e33e6854d805" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 15:29:03.919867 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:03.919827 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" event={"ID":"5a896388-6490-47c0-9eed-5c81fde73165","Type":"ContainerStarted","Data":"c6065e0fedecb88b3e2893d8dcfeabc040d6e02612fa8c72c99332991a862f68"} Nov 03 15:29:04.104783 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:04.104725 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" podStartSLOduration=3.104689438 podStartE2EDuration="3.104689438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:04.096291875 +0000 UTC m=+1251.527249686" watchObservedRunningTime="2026-11-03 15:29:04.104689438 +0000 UTC m=+1251.535647244" Nov 03 15:29:04.299134 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:04.299039 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:04.300261 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:04.300235 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 15:29:04.300341 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:04.300281 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" podUID="5a896388-6490-47c0-9eed-5c81fde73165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 15:29:04.321601 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:04.321551 4290 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:04.321819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:04.321630 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" podUID="6f7d95e8-3d59-481b-bdfd-e33e6854d805" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 15:29:05.299137 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:05.299096 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 15:29:05.299331 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:05.299168 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" podUID="5a896388-6490-47c0-9eed-5c81fde73165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 15:29:05.321976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:05.321940 4290 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:05.322341 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:05.322003 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" podUID="6f7d95e8-3d59-481b-bdfd-e33e6854d805" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 15:29:06.299547 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:06.299506 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 15:29:06.299780 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:06.299579 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" podUID="5a896388-6490-47c0-9eed-5c81fde73165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 15:29:06.321828 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:06.321794 4290 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:06.321981 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:06.321859 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" podUID="6f7d95e8-3d59-481b-bdfd-e33e6854d805" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 15:29:07.299413 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:07.299369 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 15:29:07.299845 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:07.299432 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" podUID="5a896388-6490-47c0-9eed-5c81fde73165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 15:29:07.322184 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:07.322147 4290 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:07.322316 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:07.322199 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" podUID="6f7d95e8-3d59-481b-bdfd-e33e6854d805" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 15:29:08.299380 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.299328 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 15:29:08.299588 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.299399 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" podUID="5a896388-6490-47c0-9eed-5c81fde73165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 15:29:08.321886 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.321854 4290 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:08.322096 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.321910 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" podUID="6f7d95e8-3d59-481b-bdfd-e33e6854d805" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 15:29:08.593639 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.593546 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw"] Nov 03 15:29:08.599770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.599743 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.633723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.633653 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpwm2\" (UniqueName: \"kubernetes.io/projected/2b9d9187-4d88-4a6b-bd10-c29febff43bd-kube-api-access-gpwm2\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.634020 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.633997 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-home\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.634200 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.634181 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9d9187-4d88-4a6b-bd10-c29febff43bd-tls-certs\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.634397 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.634380 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-model-cache\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.634630 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.634612 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-dshm\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.634782 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.634765 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-tmp-dir\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.655283 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.655247 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-34e8c5\"/\"distinct-2-w5-34e8c5-kserve-self-signed-certs\"" Nov 03 15:29:08.703004 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.702930 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw"] Nov 03 15:29:08.751176 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.736252 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-dshm\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.751176 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.736309 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-tmp-dir\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.751176 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.736460 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpwm2\" (UniqueName: \"kubernetes.io/projected/2b9d9187-4d88-4a6b-bd10-c29febff43bd-kube-api-access-gpwm2\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.751176 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.736498 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-home\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.751176 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.736550 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9d9187-4d88-4a6b-bd10-c29febff43bd-tls-certs\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.751176 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.736648 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-model-cache\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.751176 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.738120 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-tmp-dir\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.779202 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.778970 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-model-cache\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.807952 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.806915 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-home\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.810814 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:29:08.810776 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:29:08.813217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.812905 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpwm2\" (UniqueName: \"kubernetes.io/projected/2b9d9187-4d88-4a6b-bd10-c29febff43bd-kube-api-access-gpwm2\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.813871 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.813824 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-dshm\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.814036 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.814011 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9d9187-4d88-4a6b-bd10-c29febff43bd-tls-certs\") pod \"distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:08.922636 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:08.922592 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:09.299534 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:09.299434 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 15:29:09.299819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:09.299787 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" podUID="5a896388-6490-47c0-9eed-5c81fde73165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 15:29:09.471857 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:09.471826 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw"] Nov 03 15:29:09.910426 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:09.910362 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:09.912365 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:09.912313 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:09.913080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:09.913058 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:29:10.026802 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:10.026763 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" event={"ID":"2b9d9187-4d88-4a6b-bd10-c29febff43bd","Type":"ContainerStarted","Data":"ab85753f61ff2270f809e1dd859bb2adb9d1c9333c89414787e283eb57a89b12"} Nov 03 15:29:10.026981 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:10.026810 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" event={"ID":"2b9d9187-4d88-4a6b-bd10-c29febff43bd","Type":"ContainerStarted","Data":"83103dcdb885b2b1ea2053e0fa9069159172f210be3e30468aefd38874a28b16"} Nov 03 15:29:10.278672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:10.278560 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podStartSLOduration=2.278538973 podStartE2EDuration="2.278538973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:10.275292016 +0000 UTC m=+1257.706249818" watchObservedRunningTime="2026-11-03 15:29:10.278538973 +0000 UTC m=+1257.709496780" Nov 03 15:29:10.299119 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:10.299086 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 15:29:10.299286 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:10.299141 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" podUID="5a896388-6490-47c0-9eed-5c81fde73165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 15:29:10.924765 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:10.924693 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:11.304288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:11.304206 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:11.304534 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:11.304511 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:11.305411 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:11.305371 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:29:18.944764 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:18.933825 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:19.083729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:19.080639 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:19.208592 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:19.208214 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:31.137595 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:31.137557 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh"] Nov 03 15:29:31.143480 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:31.143458 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" Nov 03 15:29:31.202573 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:31.202536 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh"] Nov 03 15:29:31.244928 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:31.244891 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:29:31.245109 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:31.245000 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:29:31.304540 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:31.304502 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lkqd\" (UniqueName: \"kubernetes.io/projected/03d86200-4c4d-4be5-a003-4dee8e1f974f-kube-api-access-2lkqd\") pod \"maas-api-revoke-sub-2615d260-hrvnh\" (UID: \"03d86200-4c4d-4be5-a003-4dee8e1f974f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" Nov 03 15:29:31.304759 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:31.304551 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03d86200-4c4d-4be5-a003-4dee8e1f974f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2615d260-hrvnh\" (UID: \"03d86200-4c4d-4be5-a003-4dee8e1f974f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" Nov 03 15:29:31.304759 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:31.304721 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03d86200-4c4d-4be5-a003-4dee8e1f974f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2615d260-hrvnh\" (UID: \"03d86200-4c4d-4be5-a003-4dee8e1f974f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" Nov 03 15:29:31.406220 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:31.406130 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lkqd\" (UniqueName: \"kubernetes.io/projected/03d86200-4c4d-4be5-a003-4dee8e1f974f-kube-api-access-2lkqd\") pod \"maas-api-revoke-sub-2615d260-hrvnh\" (UID: \"03d86200-4c4d-4be5-a003-4dee8e1f974f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" Nov 03 15:29:31.406220 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:31.406184 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03d86200-4c4d-4be5-a003-4dee8e1f974f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2615d260-hrvnh\" (UID: \"03d86200-4c4d-4be5-a003-4dee8e1f974f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" Nov 03 15:29:31.406412 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:31.406262 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03d86200-4c4d-4be5-a003-4dee8e1f974f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2615d260-hrvnh\" (UID: \"03d86200-4c4d-4be5-a003-4dee8e1f974f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" Nov 03 15:29:31.406908 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:31.406883 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03d86200-4c4d-4be5-a003-4dee8e1f974f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2615d260-hrvnh\" (UID: \"03d86200-4c4d-4be5-a003-4dee8e1f974f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" Nov 03 15:29:31.409058 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:31.409033 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03d86200-4c4d-4be5-a003-4dee8e1f974f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2615d260-hrvnh\" (UID: \"03d86200-4c4d-4be5-a003-4dee8e1f974f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" Nov 03 15:29:31.468641 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:31.468603 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lkqd\" (UniqueName: \"kubernetes.io/projected/03d86200-4c4d-4be5-a003-4dee8e1f974f-kube-api-access-2lkqd\") pod \"maas-api-revoke-sub-2615d260-hrvnh\" (UID: \"03d86200-4c4d-4be5-a003-4dee8e1f974f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" Nov 03 15:29:31.757557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:31.757481 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" Nov 03 15:29:32.157941 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:29:32.157905 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d86200_4c4d_4be5_a003_4dee8e1f974f.slice/crio-6af0cde18e4edffabb5635b7f3a1d5801c881b8a15e3ecc00264013933246187 WatchSource:0}: Error finding container 6af0cde18e4edffabb5635b7f3a1d5801c881b8a15e3ecc00264013933246187: Status 404 returned error can't find the container with id 6af0cde18e4edffabb5635b7f3a1d5801c881b8a15e3ecc00264013933246187 Nov 03 15:29:32.159438 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:32.159412 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh"] Nov 03 15:29:32.305430 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:32.305396 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" event={"ID":"03d86200-4c4d-4be5-a003-4dee8e1f974f","Type":"ContainerStarted","Data":"6af0cde18e4edffabb5635b7f3a1d5801c881b8a15e3ecc00264013933246187"} Nov 03 15:29:33.314988 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:33.314956 4290 generic.go:356] "Generic (PLEG): container finished" podID="03d86200-4c4d-4be5-a003-4dee8e1f974f" containerID="ef956686ce6751053f59ff4d560d09bae78067e78434603e2d13155ea56e6a8d" exitCode=0 Nov 03 15:29:33.315356 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:33.315045 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" event={"ID":"03d86200-4c4d-4be5-a003-4dee8e1f974f","Type":"ContainerDied","Data":"ef956686ce6751053f59ff4d560d09bae78067e78434603e2d13155ea56e6a8d"} Nov 03 15:29:34.491309 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:34.491283 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" Nov 03 15:29:34.650170 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:34.650133 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03d86200-4c4d-4be5-a003-4dee8e1f974f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03d86200-4c4d-4be5-a003-4dee8e1f974f-maas-api-openshift-ca\") pod \"03d86200-4c4d-4be5-a003-4dee8e1f974f\" (UID: \"03d86200-4c4d-4be5-a003-4dee8e1f974f\") " Nov 03 15:29:34.650367 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:34.650237 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03d86200-4c4d-4be5-a003-4dee8e1f974f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03d86200-4c4d-4be5-a003-4dee8e1f974f-maas-api-xks-ca\") pod \"03d86200-4c4d-4be5-a003-4dee8e1f974f\" (UID: \"03d86200-4c4d-4be5-a003-4dee8e1f974f\") " Nov 03 15:29:34.650367 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:34.650299 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d86200-4c4d-4be5-a003-4dee8e1f974f-kube-api-access-2lkqd\" (UniqueName: \"kubernetes.io/projected/03d86200-4c4d-4be5-a003-4dee8e1f974f-kube-api-access-2lkqd\") pod \"03d86200-4c4d-4be5-a003-4dee8e1f974f\" (UID: \"03d86200-4c4d-4be5-a003-4dee8e1f974f\") " Nov 03 15:29:34.650577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:34.650548 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03d86200-4c4d-4be5-a003-4dee8e1f974f-maas-api-openshift-ca" pod "03d86200-4c4d-4be5-a003-4dee8e1f974f" (UID: "03d86200-4c4d-4be5-a003-4dee8e1f974f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:29:34.651022 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:34.651000 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03d86200-4c4d-4be5-a003-4dee8e1f974f-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:34.653248 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:34.653217 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d86200-4c4d-4be5-a003-4dee8e1f974f-kube-api-access-2lkqd" pod "03d86200-4c4d-4be5-a003-4dee8e1f974f" (UID: "03d86200-4c4d-4be5-a003-4dee8e1f974f"). InnerVolumeSpecName "kube-api-access-2lkqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:29:34.653498 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:34.653483 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03d86200-4c4d-4be5-a003-4dee8e1f974f-maas-api-xks-ca" pod "03d86200-4c4d-4be5-a003-4dee8e1f974f" (UID: "03d86200-4c4d-4be5-a003-4dee8e1f974f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:29:34.752291 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:34.752259 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03d86200-4c4d-4be5-a003-4dee8e1f974f-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:34.752291 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:34.752286 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lkqd\" (UniqueName: \"kubernetes.io/projected/03d86200-4c4d-4be5-a003-4dee8e1f974f-kube-api-access-2lkqd\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:35.333102 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:35.333078 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" Nov 03 15:29:35.333365 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:35.333123 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh" event={"ID":"03d86200-4c4d-4be5-a003-4dee8e1f974f","Type":"ContainerDied","Data":"6af0cde18e4edffabb5635b7f3a1d5801c881b8a15e3ecc00264013933246187"} Nov 03 15:29:35.333365 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:35.333169 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6af0cde18e4edffabb5635b7f3a1d5801c881b8a15e3ecc00264013933246187" Nov 03 15:29:38.704052 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:29:38.704016 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:29:46.029878 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.029833 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q"] Nov 03 15:29:46.030552 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.030537 4290 cpu_manager.go:414] "containerMap: removing container" podUID="03d86200-4c4d-4be5-a003-4dee8e1f974f" containerName="revoke-keys" Nov 03 15:29:46.030598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.030554 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d86200-4c4d-4be5-a003-4dee8e1f974f" containerName="revoke-keys" Nov 03 15:29:46.030735 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.030724 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d86200-4c4d-4be5-a003-4dee8e1f974f" containerName="revoke-keys" Nov 03 15:29:46.035848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.035829 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.108244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.108207 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.108244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.108247 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.108454 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.108372 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.108454 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.108403 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.108525 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.108496 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.108558 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.108546 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.108594 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.108565 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.108652 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.108627 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.108756 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.108743 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdmq9\" (UniqueName: \"kubernetes.io/projected/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-kube-api-access-hdmq9\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.137246 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.137210 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q"] Nov 03 15:29:46.210093 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.210054 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmq9\" (UniqueName: \"kubernetes.io/projected/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-kube-api-access-hdmq9\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.210290 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.210166 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.210290 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.210193 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.210290 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.210263 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.210484 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.210303 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.210484 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.210367 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.210484 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.210432 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.210484 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.210462 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.210794 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.210491 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.210989 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.210964 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.211353 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.211330 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.211753 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.211681 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.211876 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.211781 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.212187 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.212164 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.213895 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.213874 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.214889 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.214870 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.276861 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.276825 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.278055 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.278030 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdmq9\" (UniqueName: \"kubernetes.io/projected/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-kube-api-access-hdmq9\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.350692 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.350589 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:46.725597 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.725562 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj"] Nov 03 15:29:46.731586 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.731559 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.793592 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.793556 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj"] Nov 03 15:29:46.822525 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.822489 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.822525 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.822525 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.822812 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.822584 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.822812 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.822619 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.822812 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.822691 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vscxf\" (UniqueName: \"kubernetes.io/projected/b4e58dd3-dccf-45da-93fd-e4bbabf337db-kube-api-access-vscxf\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.822812 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.822736 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.823085 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.823053 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.823167 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.823147 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.823387 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.823322 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.864725 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.864229 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q"] Nov 03 15:29:46.893061 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:29:46.893032 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a18e3ac_9d85_4ac3_a1dd_310fbbe263e9.slice/crio-526f1352329e5c03ec9ed6a747c5237b5df0a0a6e338c59e16c38f0843bc9613 WatchSource:0}: Error finding container 526f1352329e5c03ec9ed6a747c5237b5df0a0a6e338c59e16c38f0843bc9613: Status 404 returned error can't find the container with id 526f1352329e5c03ec9ed6a747c5237b5df0a0a6e338c59e16c38f0843bc9613 Nov 03 15:29:46.895599 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.895565 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:29:46.895675 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.895634 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:29:46.895675 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.895665 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:29:46.924879 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.924852 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vscxf\" (UniqueName: \"kubernetes.io/projected/b4e58dd3-dccf-45da-93fd-e4bbabf337db-kube-api-access-vscxf\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.924983 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.924881 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.924983 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.924946 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.924983 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.924966 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.925146 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.925024 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.925146 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.925088 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.925146 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.925111 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.925310 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.925186 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.925310 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.925233 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.925406 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.925384 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.925504 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.925483 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.925567 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.925508 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.925622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.925571 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.925771 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.925750 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.927657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.927637 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.927825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.927804 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:46.988466 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:46.988393 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vscxf\" (UniqueName: \"kubernetes.io/projected/b4e58dd3-dccf-45da-93fd-e4bbabf337db-kube-api-access-vscxf\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:47.005192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:47.005162 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:47.049550 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:47.049464 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:47.440142 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:47.440092 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" event={"ID":"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9","Type":"ContainerStarted","Data":"970553885c9dbec874bdf27dbf8f8707faec4c819e96b33205da59251f352671"} Nov 03 15:29:47.440414 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:47.440204 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" event={"ID":"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9","Type":"ContainerStarted","Data":"526f1352329e5c03ec9ed6a747c5237b5df0a0a6e338c59e16c38f0843bc9613"} Nov 03 15:29:47.510747 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:29:47.510682 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e58dd3_dccf_45da_93fd_e4bbabf337db.slice/crio-55d6bd77ad20a3a06c33921b7c3c2dc5d05ed8084606d24f69deb84dffd49da2 WatchSource:0}: Error finding container 55d6bd77ad20a3a06c33921b7c3c2dc5d05ed8084606d24f69deb84dffd49da2: Status 404 returned error can't find the container with id 55d6bd77ad20a3a06c33921b7c3c2dc5d05ed8084606d24f69deb84dffd49da2 Nov 03 15:29:47.513904 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:47.513870 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:29:47.513997 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:47.513942 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:29:47.513997 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:47.513973 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:29:47.556321 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:47.556280 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj"] Nov 03 15:29:47.566809 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:47.566673 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" podStartSLOduration=2.566655323 podStartE2EDuration="2.566655323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:47.564058842 +0000 UTC m=+1294.995016649" watchObservedRunningTime="2026-11-03 15:29:47.566655323 +0000 UTC m=+1294.997613130" Nov 03 15:29:48.357728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:48.355061 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:48.369728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:48.367944 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 15:29:48.369728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:48.368025 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" podUID="5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 15:29:48.483773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:48.482584 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" event={"ID":"b4e58dd3-dccf-45da-93fd-e4bbabf337db","Type":"ContainerStarted","Data":"79f5c7f5d8852e9341030a17f4b5ca22a9d1c8d56e269fff1dcbfe891ab7a45e"} Nov 03 15:29:48.483773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:48.482640 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" event={"ID":"b4e58dd3-dccf-45da-93fd-e4bbabf337db","Type":"ContainerStarted","Data":"55d6bd77ad20a3a06c33921b7c3c2dc5d05ed8084606d24f69deb84dffd49da2"} Nov 03 15:29:48.730751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:48.730675 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" podStartSLOduration=2.730661456 podStartE2EDuration="2.730661456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:48.676875214 +0000 UTC m=+1296.107833024" watchObservedRunningTime="2026-11-03 15:29:48.730661456 +0000 UTC m=+1296.161619262" Nov 03 15:29:49.064197 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:49.064098 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:49.080886 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:49.080847 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 15:29:49.081177 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:49.081142 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" podUID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 15:29:49.359910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:49.353885 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 15:29:49.359910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:49.353941 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" podUID="5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 15:29:50.068863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:50.068826 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 15:29:50.069142 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:50.069113 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" podUID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 15:29:50.354782 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:50.353163 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 15:29:50.354782 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:50.353254 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" podUID="5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 15:29:51.071730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:51.059138 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 15:29:51.071730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:51.059231 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" podUID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 15:29:51.362833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:51.361809 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 15:29:51.362833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:51.361889 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" podUID="5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 15:29:52.060268 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:52.060229 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 15:29:52.060595 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:52.060561 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" podUID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 15:29:52.194853 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:52.194798 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:52.195396 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:52.195236 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 15:29:52.373826 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:52.373792 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 15:29:52.374125 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:52.374097 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" podUID="5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 15:29:52.413834 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:52.413786 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:52.583262 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:52.583220 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:52.650804 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:52.650329 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Nov 03 15:29:52.733728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:52.728601 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:52.786048 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:52.784844 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 15:29:52.879863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:52.879807 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:53.026908 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.026804 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:53.052125 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.052086 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 15:29:53.058832 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.052371 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" podUID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 15:29:53.251729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.228514 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:53.251729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.228935 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:53.283792 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.283332 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:29:53.288723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.285122 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:53.329352 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.312866 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 15:29:53.416454 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.416412 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 15:29:53.416763 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.416731 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" podUID="5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 15:29:53.439728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.419161 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 15:29:53.528215 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.528182 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:29:53.629499 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.620787 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 15:29:53.629499 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.628289 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:29:53.652913 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.652868 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:53.799810 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.799758 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:29:53.814736 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.802534 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:53.837804 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.837773 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:29:53.882167 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.881782 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:53.959729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:53.941035 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:29:54.057297 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:54.057266 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 15:29:54.057611 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:54.057563 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" podUID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 15:29:54.251777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:54.251657 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:54.283233 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:54.269800 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:29:54.283233 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:54.269870 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" podUID="12553958-5457-4c66-b011-431e954fef89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:54.288888 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:54.288836 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:54.340891 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:54.340845 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 15:29:54.373728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:54.345131 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:29:54.472910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:54.472859 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 15:29:54.482879 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:54.482798 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" podUID="5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 15:29:54.552815 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:54.552445 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:54.621031 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:54.620989 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 15:29:54.736730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:54.735960 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:54.923866 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:54.910221 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:54.994727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:54.910831 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:55.060772 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:55.056192 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:29:55.075144 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:55.075112 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 15:29:55.088039 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:55.087904 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" podUID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 15:29:55.295509 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:55.294825 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:55.344456 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:55.344429 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:29:55.346159 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:55.346057 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:29:55.368357 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:55.367117 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 15:29:55.395724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:55.393667 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:29:55.422997 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:55.422962 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 15:29:55.423280 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:55.423254 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" podUID="5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 15:29:55.486731 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:55.438808 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:29:55.515443 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:55.515413 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:29:55.519547 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:55.519525 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:29:55.536743 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:55.532842 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:29:56.050123 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:56.050083 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 15:29:56.050327 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:56.050144 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" podUID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 15:29:56.351977 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:56.351890 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 15:29:56.351977 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:56.351946 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" podUID="5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 15:29:57.052934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:57.051062 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 15:29:57.052934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:57.051124 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" podUID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 15:29:57.379438 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:57.379410 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:57.380775 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:57.380752 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:57.382289 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:57.382256 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:29:59.090951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:59.090876 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:29:59.091467 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:59.090966 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" podUID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:59.320211 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:59.320168 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:59.321404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:59.321379 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:29:59.322087 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:29:59.322069 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:30:00.226264 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.226228 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw"] Nov 03 15:30:00.232125 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.232103 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4"] Nov 03 15:30:00.232254 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.232228 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" Nov 03 15:30:00.236617 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.236597 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" Nov 03 15:30:00.258227 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.258196 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw\" (UID: \"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" Nov 03 15:30:00.258471 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.258453 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz56b\" (UniqueName: \"kubernetes.io/projected/dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee-kube-api-access-mz56b\") pod \"maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw\" (UID: \"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" Nov 03 15:30:00.276534 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.276506 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9"] Nov 03 15:30:00.282132 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.282111 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz"] Nov 03 15:30:00.282247 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.282221 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" Nov 03 15:30:00.284681 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.284658 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:30:00.286560 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.286541 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" Nov 03 15:30:00.305993 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.305966 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw"] Nov 03 15:30:00.322747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.321879 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4"] Nov 03 15:30:00.354601 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.354572 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9"] Nov 03 15:30:00.365654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.363779 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ba8f6e-c38e-4072-b776-a5269ef2f7d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4\" (UID: \"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" Nov 03 15:30:00.365654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.363991 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz56b\" (UniqueName: \"kubernetes.io/projected/dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee-kube-api-access-mz56b\") pod \"maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw\" (UID: \"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" Nov 03 15:30:00.365654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.364027 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ec8782-7966-4c2d-929a-197fb81ad397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e0fe64-xzzh9\" (UID: \"76ec8782-7966-4c2d-929a-197fb81ad397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" Nov 03 15:30:00.365654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.364050 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ec8782-7966-4c2d-929a-197fb81ad397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e0fe64-xzzh9\" (UID: \"76ec8782-7966-4c2d-929a-197fb81ad397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" Nov 03 15:30:00.365654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.364116 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rxqf\" (UniqueName: \"kubernetes.io/projected/8bde81d0-b4d2-4393-b052-e550d9241d2e-kube-api-access-8rxqf\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz\" (UID: \"8bde81d0-b4d2-4393-b052-e550d9241d2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" Nov 03 15:30:00.365654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.364190 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw\" (UID: \"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" Nov 03 15:30:00.365654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.364247 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bde81d0-b4d2-4393-b052-e550d9241d2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz\" (UID: \"8bde81d0-b4d2-4393-b052-e550d9241d2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" Nov 03 15:30:00.365654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.364297 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj4mq\" (UniqueName: \"kubernetes.io/projected/d5ba8f6e-c38e-4072-b776-a5269ef2f7d0-kube-api-access-hj4mq\") pod \"maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4\" (UID: \"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" Nov 03 15:30:00.365654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.364325 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxlf\" (UniqueName: \"kubernetes.io/projected/76ec8782-7966-4c2d-929a-197fb81ad397-kube-api-access-cpxlf\") pod \"maas-api-revoke-sub-04e0fe64-xzzh9\" (UID: \"76ec8782-7966-4c2d-929a-197fb81ad397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" Nov 03 15:30:00.365654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.365535 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw\" (UID: \"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" Nov 03 15:30:00.374720 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.374672 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz"] Nov 03 15:30:00.383464 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.383439 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:30:00.426594 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.426565 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz56b\" (UniqueName: \"kubernetes.io/projected/dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee-kube-api-access-mz56b\") pod \"maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw\" (UID: \"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" Nov 03 15:30:00.465353 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.465319 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bde81d0-b4d2-4393-b052-e550d9241d2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz\" (UID: \"8bde81d0-b4d2-4393-b052-e550d9241d2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" Nov 03 15:30:00.465628 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.465597 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj4mq\" (UniqueName: \"kubernetes.io/projected/d5ba8f6e-c38e-4072-b776-a5269ef2f7d0-kube-api-access-hj4mq\") pod \"maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4\" (UID: \"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" Nov 03 15:30:00.465828 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.465644 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxlf\" (UniqueName: \"kubernetes.io/projected/76ec8782-7966-4c2d-929a-197fb81ad397-kube-api-access-cpxlf\") pod \"maas-api-revoke-sub-04e0fe64-xzzh9\" (UID: \"76ec8782-7966-4c2d-929a-197fb81ad397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" Nov 03 15:30:00.465828 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.465747 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ba8f6e-c38e-4072-b776-a5269ef2f7d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4\" (UID: \"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" Nov 03 15:30:00.466504 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.466005 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ec8782-7966-4c2d-929a-197fb81ad397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e0fe64-xzzh9\" (UID: \"76ec8782-7966-4c2d-929a-197fb81ad397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" Nov 03 15:30:00.466504 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.466037 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ec8782-7966-4c2d-929a-197fb81ad397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e0fe64-xzzh9\" (UID: \"76ec8782-7966-4c2d-929a-197fb81ad397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" Nov 03 15:30:00.466504 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.466087 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bde81d0-b4d2-4393-b052-e550d9241d2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz\" (UID: \"8bde81d0-b4d2-4393-b052-e550d9241d2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" Nov 03 15:30:00.466504 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.466107 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rxqf\" (UniqueName: \"kubernetes.io/projected/8bde81d0-b4d2-4393-b052-e550d9241d2e-kube-api-access-8rxqf\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz\" (UID: \"8bde81d0-b4d2-4393-b052-e550d9241d2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" Nov 03 15:30:00.466995 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.466966 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ec8782-7966-4c2d-929a-197fb81ad397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e0fe64-xzzh9\" (UID: \"76ec8782-7966-4c2d-929a-197fb81ad397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" Nov 03 15:30:00.467071 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.467016 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ba8f6e-c38e-4072-b776-a5269ef2f7d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4\" (UID: \"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" Nov 03 15:30:00.468968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.468945 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ec8782-7966-4c2d-929a-197fb81ad397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e0fe64-xzzh9\" (UID: \"76ec8782-7966-4c2d-929a-197fb81ad397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" Nov 03 15:30:00.527152 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.527087 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj4mq\" (UniqueName: \"kubernetes.io/projected/d5ba8f6e-c38e-4072-b776-a5269ef2f7d0-kube-api-access-hj4mq\") pod \"maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4\" (UID: \"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" Nov 03 15:30:00.527293 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.527274 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rxqf\" (UniqueName: \"kubernetes.io/projected/8bde81d0-b4d2-4393-b052-e550d9241d2e-kube-api-access-8rxqf\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz\" (UID: \"8bde81d0-b4d2-4393-b052-e550d9241d2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" Nov 03 15:30:00.527461 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.527437 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxlf\" (UniqueName: \"kubernetes.io/projected/76ec8782-7966-4c2d-929a-197fb81ad397-kube-api-access-cpxlf\") pod \"maas-api-revoke-sub-04e0fe64-xzzh9\" (UID: \"76ec8782-7966-4c2d-929a-197fb81ad397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" Nov 03 15:30:00.551328 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.551293 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" Nov 03 15:30:00.557659 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.557630 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" Nov 03 15:30:00.608792 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.608750 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" Nov 03 15:30:00.611885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.611854 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" Nov 03 15:30:00.971232 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:30:00.971199 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc74c57_6ad7_464b_a9fd_b1eebeaf1dee.slice/crio-266590fd941ec10553b23fa7d6adb4cb3991e57f4155d9f53e5472b836cf090b WatchSource:0}: Error finding container 266590fd941ec10553b23fa7d6adb4cb3991e57f4155d9f53e5472b836cf090b: Status 404 returned error can't find the container with id 266590fd941ec10553b23fa7d6adb4cb3991e57f4155d9f53e5472b836cf090b Nov 03 15:30:00.971431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:00.971410 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw"] Nov 03 15:30:01.022097 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:30:01.022065 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ba8f6e_c38e_4072_b776_a5269ef2f7d0.slice/crio-7c727f38bf16fd4022787897dbf06b4e1a67bf6e2d097ce3a33ea5f0d21fbff3 WatchSource:0}: Error finding container 7c727f38bf16fd4022787897dbf06b4e1a67bf6e2d097ce3a33ea5f0d21fbff3: Status 404 returned error can't find the container with id 7c727f38bf16fd4022787897dbf06b4e1a67bf6e2d097ce3a33ea5f0d21fbff3 Nov 03 15:30:01.022754 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:01.022732 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4"] Nov 03 15:30:01.077573 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:01.077544 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz"] Nov 03 15:30:01.082141 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:01.082113 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9"] Nov 03 15:30:01.143057 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:30:01.143020 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bde81d0_b4d2_4393_b052_e550d9241d2e.slice/crio-6adb86e0da93de727802e58e7fb3aad4aacf82cf9259cf1703dd60b4ceab44c7 WatchSource:0}: Error finding container 6adb86e0da93de727802e58e7fb3aad4aacf82cf9259cf1703dd60b4ceab44c7: Status 404 returned error can't find the container with id 6adb86e0da93de727802e58e7fb3aad4aacf82cf9259cf1703dd60b4ceab44c7 Nov 03 15:30:01.144641 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:30:01.144534 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ec8782_7966_4c2d_929a_197fb81ad397.slice/crio-54b2139da18b42c1e14203d9ec6a9f47140cec6bda4946fc45f38813e6e07160 WatchSource:0}: Error finding container 54b2139da18b42c1e14203d9ec6a9f47140cec6bda4946fc45f38813e6e07160: Status 404 returned error can't find the container with id 54b2139da18b42c1e14203d9ec6a9f47140cec6bda4946fc45f38813e6e07160 Nov 03 15:30:01.417853 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:01.417819 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" event={"ID":"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0","Type":"ContainerStarted","Data":"7c727f38bf16fd4022787897dbf06b4e1a67bf6e2d097ce3a33ea5f0d21fbff3"} Nov 03 15:30:01.419353 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:01.419330 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" event={"ID":"8bde81d0-b4d2-4393-b052-e550d9241d2e","Type":"ContainerStarted","Data":"6adb86e0da93de727802e58e7fb3aad4aacf82cf9259cf1703dd60b4ceab44c7"} Nov 03 15:30:01.420621 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:01.420602 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" event={"ID":"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee","Type":"ContainerStarted","Data":"266590fd941ec10553b23fa7d6adb4cb3991e57f4155d9f53e5472b836cf090b"} Nov 03 15:30:01.421866 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:01.421839 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" event={"ID":"76ec8782-7966-4c2d-929a-197fb81ad397","Type":"ContainerStarted","Data":"54b2139da18b42c1e14203d9ec6a9f47140cec6bda4946fc45f38813e6e07160"} Nov 03 15:30:02.151883 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.151853 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4"] Nov 03 15:30:02.157173 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.157144 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.206375 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.206339 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.206543 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.206401 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.206587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.206531 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.206677 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.206654 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.206773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.206695 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srm69\" (UniqueName: \"kubernetes.io/projected/28b9fcb2-e588-4d76-909c-244704b9a658-kube-api-access-srm69\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.206773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.206743 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b9fcb2-e588-4d76-909c-244704b9a658-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.206939 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.206917 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b9fcb2-e588-4d76-909c-244704b9a658-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.206985 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.206974 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.207099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.207081 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b9fcb2-e588-4d76-909c-244704b9a658-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.219766 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.219734 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4"] Nov 03 15:30:02.308976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.308940 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.309122 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.309003 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.309122 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.309044 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.309122 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.309110 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.309285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.309145 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srm69\" (UniqueName: \"kubernetes.io/projected/28b9fcb2-e588-4d76-909c-244704b9a658-kube-api-access-srm69\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.309285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.309178 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b9fcb2-e588-4d76-909c-244704b9a658-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.309397 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.309367 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.309477 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.309395 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b9fcb2-e588-4d76-909c-244704b9a658-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.309477 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.309438 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.309477 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.309458 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.309641 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.309539 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b9fcb2-e588-4d76-909c-244704b9a658-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.309728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.309458 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.309974 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.309951 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.310080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.310061 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b9fcb2-e588-4d76-909c-244704b9a658-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.311601 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.311578 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.311762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.311745 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b9fcb2-e588-4d76-909c-244704b9a658-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.371484 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.371455 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b9fcb2-e588-4d76-909c-244704b9a658-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.371622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.371568 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srm69\" (UniqueName: \"kubernetes.io/projected/28b9fcb2-e588-4d76-909c-244704b9a658-kube-api-access-srm69\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.433091 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.433013 4290 generic.go:356] "Generic (PLEG): container finished" podID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" containerID="70c0e709cea24040d8f74f6573e6b6622c6a0266d986f07237f8dadd7f94876e" exitCode=60 Nov 03 15:30:02.433478 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.433096 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" event={"ID":"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee","Type":"ContainerDied","Data":"70c0e709cea24040d8f74f6573e6b6622c6a0266d986f07237f8dadd7f94876e"} Nov 03 15:30:02.433478 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.433375 4290 scope.go:122] "RemoveContainer" containerID="70c0e709cea24040d8f74f6573e6b6622c6a0266d986f07237f8dadd7f94876e" Nov 03 15:30:02.437223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.435216 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" event={"ID":"76ec8782-7966-4c2d-929a-197fb81ad397","Type":"ContainerStarted","Data":"3ff05e8de2e77f92b702f8252e21cacfa03ee9d5c4e5df7f0e1778d904d7cbab"} Nov 03 15:30:02.437223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.437178 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" event={"ID":"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0","Type":"ContainerDied","Data":"c2e1482fb69911bb2fe307b09e6be4fa19817473bc4cefa024c371238814c9c4"} Nov 03 15:30:02.437223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.436960 4290 generic.go:356] "Generic (PLEG): container finished" podID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" containerID="c2e1482fb69911bb2fe307b09e6be4fa19817473bc4cefa024c371238814c9c4" exitCode=60 Nov 03 15:30:02.437437 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.437238 4290 scope.go:122] "RemoveContainer" containerID="c2e1482fb69911bb2fe307b09e6be4fa19817473bc4cefa024c371238814c9c4" Nov 03 15:30:02.439450 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.439430 4290 generic.go:356] "Generic (PLEG): container finished" podID="8bde81d0-b4d2-4393-b052-e550d9241d2e" containerID="32786c22655e139450672460f49ee67eec062bcac5212b0bd171f69c6fc0fe42" exitCode=60 Nov 03 15:30:02.439561 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.439488 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" event={"ID":"8bde81d0-b4d2-4393-b052-e550d9241d2e","Type":"ContainerDied","Data":"32786c22655e139450672460f49ee67eec062bcac5212b0bd171f69c6fc0fe42"} Nov 03 15:30:02.439767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.439751 4290 scope.go:122] "RemoveContainer" containerID="32786c22655e139450672460f49ee67eec062bcac5212b0bd171f69c6fc0fe42" Nov 03 15:30:02.522274 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.522251 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:02.904557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.904495 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" podStartSLOduration=2.904477886 podStartE2EDuration="2.904477886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:02.903729047 +0000 UTC m=+1310.334686856" watchObservedRunningTime="2026-11-03 15:30:02.904477886 +0000 UTC m=+1310.335435692" Nov 03 15:30:02.981192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:02.981169 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4"] Nov 03 15:30:03.046565 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.046009 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:03.046565 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.046100 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:03.046565 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.046147 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:03.068261 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.066509 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7319-7bca6c"] Nov 03 15:30:03.092765 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.092735 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-7bca6c" Nov 03 15:30:03.118130 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.118106 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7319-7bca6c"] Nov 03 15:30:03.146254 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.145615 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hb74t\"" Nov 03 15:30:03.224047 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.223997 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l6bk\" (UniqueName: \"kubernetes.io/projected/4b4c30ce-7197-444f-8a3e-42e46ab79dc9-kube-api-access-2l6bk\") pod \"test-curl-7319-7bca6c\" (UID: \"4b4c30ce-7197-444f-8a3e-42e46ab79dc9\") " pod="opendatahub/test-curl-7319-7bca6c" Nov 03 15:30:03.326314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.326283 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l6bk\" (UniqueName: \"kubernetes.io/projected/4b4c30ce-7197-444f-8a3e-42e46ab79dc9-kube-api-access-2l6bk\") pod \"test-curl-7319-7bca6c\" (UID: \"4b4c30ce-7197-444f-8a3e-42e46ab79dc9\") " pod="opendatahub/test-curl-7319-7bca6c" Nov 03 15:30:03.388111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.388080 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l6bk\" (UniqueName: \"kubernetes.io/projected/4b4c30ce-7197-444f-8a3e-42e46ab79dc9-kube-api-access-2l6bk\") pod \"test-curl-7319-7bca6c\" (UID: \"4b4c30ce-7197-444f-8a3e-42e46ab79dc9\") " pod="opendatahub/test-curl-7319-7bca6c" Nov 03 15:30:03.419587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.419551 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-7bca6c" Nov 03 15:30:03.458574 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.458536 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" event={"ID":"8bde81d0-b4d2-4393-b052-e550d9241d2e","Type":"ContainerStarted","Data":"651cf29f5f7d3e317f30d7f47de1c81a6fbba311c408678e9e303b3fc8cdf84b"} Nov 03 15:30:03.462779 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.462751 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" event={"ID":"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee","Type":"ContainerStarted","Data":"e494da73405d3ab3ea61a13bb18b52e9f1fef08441fa0e2eef4289680229888f"} Nov 03 15:30:03.465118 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.465088 4290 generic.go:356] "Generic (PLEG): container finished" podID="76ec8782-7966-4c2d-929a-197fb81ad397" containerID="3ff05e8de2e77f92b702f8252e21cacfa03ee9d5c4e5df7f0e1778d904d7cbab" exitCode=0 Nov 03 15:30:03.465285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.465267 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" event={"ID":"76ec8782-7966-4c2d-929a-197fb81ad397","Type":"ContainerDied","Data":"3ff05e8de2e77f92b702f8252e21cacfa03ee9d5c4e5df7f0e1778d904d7cbab"} Nov 03 15:30:03.468157 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.468136 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" event={"ID":"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0","Type":"ContainerStarted","Data":"da6163157b70ca9c10ba278549c0c69c14c48d787391ace8902cfc930edd6003"} Nov 03 15:30:03.470894 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.470866 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" event={"ID":"28b9fcb2-e588-4d76-909c-244704b9a658","Type":"ContainerStarted","Data":"6f4f017f86a40b69b4bdc4098169dea614ce8fcabd461e3f58db9cb7daa0a122"} Nov 03 15:30:03.470998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.470927 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" event={"ID":"28b9fcb2-e588-4d76-909c-244704b9a658","Type":"ContainerStarted","Data":"06cbfbe684eb44c6ce2a66ff42b9a954b3cf8627c56203fb47c532a06ee0c9cf"} Nov 03 15:30:03.523150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.522592 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:03.581043 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.580993 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" podStartSLOduration=3.580979112 podStartE2EDuration="3.580979112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:03.57524532 +0000 UTC m=+1311.006203127" watchObservedRunningTime="2026-11-03 15:30:03.580979112 +0000 UTC m=+1311.011936957" Nov 03 15:30:03.696533 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.696485 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" podStartSLOduration=3.696468918 podStartE2EDuration="3.696468918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:03.690594214 +0000 UTC m=+1311.121552021" watchObservedRunningTime="2026-11-03 15:30:03.696468918 +0000 UTC m=+1311.127426723" Nov 03 15:30:03.818110 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.818081 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7319-7bca6c"] Nov 03 15:30:03.829311 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:30:03.829244 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b4c30ce_7197_444f_8a3e_42e46ab79dc9.slice/crio-00931b9dac5b5d9db688a4646d1813b37c2f352a887c204cf0a2103b130534dc WatchSource:0}: Error finding container 00931b9dac5b5d9db688a4646d1813b37c2f352a887c204cf0a2103b130534dc: Status 404 returned error can't find the container with id 00931b9dac5b5d9db688a4646d1813b37c2f352a887c204cf0a2103b130534dc Nov 03 15:30:03.831737 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:03.831715 4290 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:30:04.012390 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.012288 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" podStartSLOduration=2.012273799 podStartE2EDuration="2.012273799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:04.010096399 +0000 UTC m=+1311.441054205" watchObservedRunningTime="2026-11-03 15:30:04.012273799 +0000 UTC m=+1311.443231605" Nov 03 15:30:04.081325 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.081287 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r"] Nov 03 15:30:04.085914 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.085877 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d706c57c-gvw2r"] Nov 03 15:30:04.135160 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.135096 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" podStartSLOduration=4.135076463 podStartE2EDuration="4.135076463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:04.131611934 +0000 UTC m=+1311.562569739" watchObservedRunningTime="2026-11-03 15:30:04.135076463 +0000 UTC m=+1311.566034273" Nov 03 15:30:04.486783 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.486747 4290 generic.go:356] "Generic (PLEG): container finished" podID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" containerID="e494da73405d3ab3ea61a13bb18b52e9f1fef08441fa0e2eef4289680229888f" exitCode=60 Nov 03 15:30:04.487304 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.486825 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" event={"ID":"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee","Type":"ContainerDied","Data":"e494da73405d3ab3ea61a13bb18b52e9f1fef08441fa0e2eef4289680229888f"} Nov 03 15:30:04.487304 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.486886 4290 scope.go:122] "RemoveContainer" containerID="70c0e709cea24040d8f74f6573e6b6622c6a0266d986f07237f8dadd7f94876e" Nov 03 15:30:04.487304 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.487173 4290 scope.go:122] "RemoveContainer" containerID="e494da73405d3ab3ea61a13bb18b52e9f1fef08441fa0e2eef4289680229888f" Nov 03 15:30:04.487545 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:04.487521 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw_odh-ai-gateway-infra(dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" podUID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" Nov 03 15:30:04.488736 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.488683 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7319-7bca6c" event={"ID":"4b4c30ce-7197-444f-8a3e-42e46ab79dc9","Type":"ContainerStarted","Data":"00931b9dac5b5d9db688a4646d1813b37c2f352a887c204cf0a2103b130534dc"} Nov 03 15:30:04.491004 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.490983 4290 generic.go:356] "Generic (PLEG): container finished" podID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" containerID="da6163157b70ca9c10ba278549c0c69c14c48d787391ace8902cfc930edd6003" exitCode=60 Nov 03 15:30:04.491115 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.491045 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" event={"ID":"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0","Type":"ContainerDied","Data":"da6163157b70ca9c10ba278549c0c69c14c48d787391ace8902cfc930edd6003"} Nov 03 15:30:04.491297 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.491283 4290 scope.go:122] "RemoveContainer" containerID="da6163157b70ca9c10ba278549c0c69c14c48d787391ace8902cfc930edd6003" Nov 03 15:30:04.491568 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:04.491548 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4_odh-ai-gateway-infra(d5ba8f6e-c38e-4072-b776-a5269ef2f7d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" podUID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" Nov 03 15:30:04.498163 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.498140 4290 generic.go:356] "Generic (PLEG): container finished" podID="8bde81d0-b4d2-4393-b052-e550d9241d2e" containerID="651cf29f5f7d3e317f30d7f47de1c81a6fbba311c408678e9e303b3fc8cdf84b" exitCode=60 Nov 03 15:30:04.498251 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.498212 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" event={"ID":"8bde81d0-b4d2-4393-b052-e550d9241d2e","Type":"ContainerDied","Data":"651cf29f5f7d3e317f30d7f47de1c81a6fbba311c408678e9e303b3fc8cdf84b"} Nov 03 15:30:04.498462 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.498447 4290 scope.go:122] "RemoveContainer" containerID="651cf29f5f7d3e317f30d7f47de1c81a6fbba311c408678e9e303b3fc8cdf84b" Nov 03 15:30:04.498836 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:04.498815 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz_odh-ai-gateway-infra(8bde81d0-b4d2-4393-b052-e550d9241d2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" podUID="8bde81d0-b4d2-4393-b052-e550d9241d2e" Nov 03 15:30:04.504222 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.504206 4290 scope.go:122] "RemoveContainer" containerID="c2e1482fb69911bb2fe307b09e6be4fa19817473bc4cefa024c371238814c9c4" Nov 03 15:30:04.519564 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.519547 4290 scope.go:122] "RemoveContainer" containerID="32786c22655e139450672460f49ee67eec062bcac5212b0bd171f69c6fc0fe42" Nov 03 15:30:04.524602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.524552 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 15:30:04.524602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.524591 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" podUID="28b9fcb2-e588-4d76-909c-244704b9a658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 15:30:04.684132 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.684101 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="332adcf3-f9d0-4c18-ab77-c358907288f3" path="/var/lib/kubelet/pods/332adcf3-f9d0-4c18-ab77-c358907288f3/volumes" Nov 03 15:30:04.734496 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.734475 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" Nov 03 15:30:04.885362 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.885325 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ec8782-7966-4c2d-929a-197fb81ad397-kube-api-access-cpxlf\" (UniqueName: \"kubernetes.io/projected/76ec8782-7966-4c2d-929a-197fb81ad397-kube-api-access-cpxlf\") pod \"76ec8782-7966-4c2d-929a-197fb81ad397\" (UID: \"76ec8782-7966-4c2d-929a-197fb81ad397\") " Nov 03 15:30:04.885559 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.885447 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ec8782-7966-4c2d-929a-197fb81ad397-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ec8782-7966-4c2d-929a-197fb81ad397-maas-api-openshift-ca\") pod \"76ec8782-7966-4c2d-929a-197fb81ad397\" (UID: \"76ec8782-7966-4c2d-929a-197fb81ad397\") " Nov 03 15:30:04.885559 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.885550 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76ec8782-7966-4c2d-929a-197fb81ad397-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ec8782-7966-4c2d-929a-197fb81ad397-maas-api-xks-ca\") pod \"76ec8782-7966-4c2d-929a-197fb81ad397\" (UID: \"76ec8782-7966-4c2d-929a-197fb81ad397\") " Nov 03 15:30:04.886006 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.885825 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ec8782-7966-4c2d-929a-197fb81ad397-maas-api-openshift-ca" pod "76ec8782-7966-4c2d-929a-197fb81ad397" (UID: "76ec8782-7966-4c2d-929a-197fb81ad397"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:04.886771 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.886688 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ec8782-7966-4c2d-929a-197fb81ad397-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:04.888577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.888550 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ec8782-7966-4c2d-929a-197fb81ad397-kube-api-access-cpxlf" pod "76ec8782-7966-4c2d-929a-197fb81ad397" (UID: "76ec8782-7966-4c2d-929a-197fb81ad397"). InnerVolumeSpecName "kube-api-access-cpxlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:04.889156 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.889123 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ec8782-7966-4c2d-929a-197fb81ad397-maas-api-xks-ca" pod "76ec8782-7966-4c2d-929a-197fb81ad397" (UID: "76ec8782-7966-4c2d-929a-197fb81ad397"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:30:04.988163 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.988123 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ec8782-7966-4c2d-929a-197fb81ad397-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:04.988163 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:04.988154 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpxlf\" (UniqueName: \"kubernetes.io/projected/76ec8782-7966-4c2d-929a-197fb81ad397-kube-api-access-cpxlf\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:05.514791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:05.514766 4290 scope.go:122] "RemoveContainer" containerID="da6163157b70ca9c10ba278549c0c69c14c48d787391ace8902cfc930edd6003" Nov 03 15:30:05.515244 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:05.515088 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4_odh-ai-gateway-infra(d5ba8f6e-c38e-4072-b776-a5269ef2f7d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" podUID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" Nov 03 15:30:05.516580 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:05.516560 4290 scope.go:122] "RemoveContainer" containerID="651cf29f5f7d3e317f30d7f47de1c81a6fbba311c408678e9e303b3fc8cdf84b" Nov 03 15:30:05.516884 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:05.516864 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz_odh-ai-gateway-infra(8bde81d0-b4d2-4393-b052-e550d9241d2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" podUID="8bde81d0-b4d2-4393-b052-e550d9241d2e" Nov 03 15:30:05.518321 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:05.518298 4290 scope.go:122] "RemoveContainer" containerID="e494da73405d3ab3ea61a13bb18b52e9f1fef08441fa0e2eef4289680229888f" Nov 03 15:30:05.518545 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:05.518516 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw_odh-ai-gateway-infra(dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" podUID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" Nov 03 15:30:05.519529 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:05.519513 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" Nov 03 15:30:05.519619 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:05.519543 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9" event={"ID":"76ec8782-7966-4c2d-929a-197fb81ad397","Type":"ContainerDied","Data":"54b2139da18b42c1e14203d9ec6a9f47140cec6bda4946fc45f38813e6e07160"} Nov 03 15:30:05.519619 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:05.519601 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54b2139da18b42c1e14203d9ec6a9f47140cec6bda4946fc45f38813e6e07160" Nov 03 15:30:05.522577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:05.522556 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 15:30:05.522672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:05.522594 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" podUID="28b9fcb2-e588-4d76-909c-244704b9a658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 15:30:06.524224 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:06.524187 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 15:30:06.524764 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:06.524247 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" podUID="28b9fcb2-e588-4d76-909c-244704b9a658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 15:30:07.522984 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:07.522951 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 15:30:07.523157 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:07.523001 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" podUID="28b9fcb2-e588-4d76-909c-244704b9a658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 15:30:08.522538 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:08.522501 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 15:30:08.522947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:08.522552 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" podUID="28b9fcb2-e588-4d76-909c-244704b9a658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 15:30:08.712782 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:08.712635 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:30:09.523084 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:09.523000 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 15:30:09.523506 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:09.523099 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" podUID="28b9fcb2-e588-4d76-909c-244704b9a658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 15:30:10.879082 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:10.879035 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:10.879630 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:10.879605 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:10.880798 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:10.880769 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:30:11.620520 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:11.620481 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7319-7bca6c" event={"ID":"4b4c30ce-7197-444f-8a3e-42e46ab79dc9","Type":"ContainerStarted","Data":"281df00009c7ec7891f7b6802ea146709e183e40dd73acea0adf31ae59e1db7f"} Nov 03 15:30:11.741045 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:11.740996 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7319-7bca6c" podStartSLOduration=1.920400947 podStartE2EDuration="8.740982399s" totalImagesPullingTime="6.820581452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:11.734928535 +0000 UTC m=+1319.165886341" watchObservedRunningTime="2026-11-03 15:30:11.740982399 +0000 UTC m=+1319.171940202" Nov 03 15:30:13.326021 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:13.325986 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7319-7bca6c"] Nov 03 15:30:13.332887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:13.332855 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7319-7bca6c"] Nov 03 15:30:13.638758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:13.638716 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7319-7bca6c" podUID="4b4c30ce-7197-444f-8a3e-42e46ab79dc9" containerName="test-curl-7319-7bca6c" containerID="cri-o://281df00009c7ec7891f7b6802ea146709e183e40dd73acea0adf31ae59e1db7f" gracePeriod=2 Nov 03 15:30:13.689931 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:13.689888 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-7bca6c\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="4b4c30ce-7197-444f-8a3e-42e46ab79dc9" pod="opendatahub/test-curl-7319-7bca6c" Nov 03 15:30:14.686364 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:14.686334 4290 scope.go:122] "RemoveContainer" containerID="e494da73405d3ab3ea61a13bb18b52e9f1fef08441fa0e2eef4289680229888f" Nov 03 15:30:15.659021 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:15.658984 4290 generic.go:356] "Generic (PLEG): container finished" podID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" containerID="e3afd4c895fe7bea91d7cd9c2eb7c42b04530b09ed1a55f994b813c90dfdaee4" exitCode=60 Nov 03 15:30:15.659144 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:15.659068 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" event={"ID":"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee","Type":"ContainerDied","Data":"e3afd4c895fe7bea91d7cd9c2eb7c42b04530b09ed1a55f994b813c90dfdaee4"} Nov 03 15:30:15.659217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:15.659157 4290 scope.go:122] "RemoveContainer" containerID="e494da73405d3ab3ea61a13bb18b52e9f1fef08441fa0e2eef4289680229888f" Nov 03 15:30:15.659553 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:15.659536 4290 scope.go:122] "RemoveContainer" containerID="e3afd4c895fe7bea91d7cd9c2eb7c42b04530b09ed1a55f994b813c90dfdaee4" Nov 03 15:30:15.659953 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:15.659935 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw_odh-ai-gateway-infra(dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" podUID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" Nov 03 15:30:15.671091 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:15.671074 4290 scope.go:122] "RemoveContainer" containerID="da6163157b70ca9c10ba278549c0c69c14c48d787391ace8902cfc930edd6003" Nov 03 15:30:15.983919 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:15.983894 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-7bca6c" Nov 03 15:30:16.081501 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.081463 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2"] Nov 03 15:30:16.082481 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.082463 4290 cpu_manager.go:414] "containerMap: removing container" podUID="76ec8782-7966-4c2d-929a-197fb81ad397" containerName="revoke-keys" Nov 03 15:30:16.082566 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.082483 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ec8782-7966-4c2d-929a-197fb81ad397" containerName="revoke-keys" Nov 03 15:30:16.082566 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.082505 4290 cpu_manager.go:414] "containerMap: removing container" podUID="4b4c30ce-7197-444f-8a3e-42e46ab79dc9" containerName="test-curl-7319-7bca6c" Nov 03 15:30:16.082566 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.082510 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4c30ce-7197-444f-8a3e-42e46ab79dc9" containerName="test-curl-7319-7bca6c" Nov 03 15:30:16.082665 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.082633 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ec8782-7966-4c2d-929a-197fb81ad397" containerName="revoke-keys" Nov 03 15:30:16.082665 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.082643 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4c30ce-7197-444f-8a3e-42e46ab79dc9" containerName="test-curl-7319-7bca6c" Nov 03 15:30:16.087166 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.087141 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.117218 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.117192 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4c30ce-7197-444f-8a3e-42e46ab79dc9-kube-api-access-2l6bk\" (UniqueName: \"kubernetes.io/projected/4b4c30ce-7197-444f-8a3e-42e46ab79dc9-kube-api-access-2l6bk\") pod \"4b4c30ce-7197-444f-8a3e-42e46ab79dc9\" (UID: \"4b4c30ce-7197-444f-8a3e-42e46ab79dc9\") " Nov 03 15:30:16.119527 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.119488 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4c30ce-7197-444f-8a3e-42e46ab79dc9-kube-api-access-2l6bk" pod "4b4c30ce-7197-444f-8a3e-42e46ab79dc9" (UID: "4b4c30ce-7197-444f-8a3e-42e46ab79dc9"). InnerVolumeSpecName "kube-api-access-2l6bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:16.145727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.145676 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2"] Nov 03 15:30:16.192949 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.190649 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-34e8c5-openshift-default-dockercfg-f8ldx\"" Nov 03 15:30:16.220361 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.220318 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.220693 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.220442 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.220693 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.220490 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.220693 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.220579 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.220693 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.220658 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.221802 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.220778 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.221802 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.220847 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.221802 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.220919 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.221802 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.221055 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqt4g\" (UniqueName: \"kubernetes.io/projected/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-kube-api-access-xqt4g\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.221802 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.221151 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l6bk\" (UniqueName: \"kubernetes.io/projected/4b4c30ce-7197-444f-8a3e-42e46ab79dc9-kube-api-access-2l6bk\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:16.322157 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.322124 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.322351 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.322180 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.322351 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.322219 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.322351 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.322285 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqt4g\" (UniqueName: \"kubernetes.io/projected/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-kube-api-access-xqt4g\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.322351 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.322338 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.322587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.322417 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.322587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.322455 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.322587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.322538 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.322744 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.322601 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.322744 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.322609 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.322844 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.322745 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.323386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.323194 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.323512 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.323453 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.324249 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.323541 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.325198 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.325170 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.325647 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.325599 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.384150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.384117 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqt4g\" (UniqueName: \"kubernetes.io/projected/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-kube-api-access-xqt4g\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.384924 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.384903 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.404713 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.404670 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:16.675105 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.675068 4290 generic.go:356] "Generic (PLEG): container finished" podID="4b4c30ce-7197-444f-8a3e-42e46ab79dc9" containerID="281df00009c7ec7891f7b6802ea146709e183e40dd73acea0adf31ae59e1db7f" exitCode=137 Nov 03 15:30:16.675288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.675173 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-7bca6c" Nov 03 15:30:16.681662 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.681633 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4c30ce-7197-444f-8a3e-42e46ab79dc9" path="/var/lib/kubelet/pods/4b4c30ce-7197-444f-8a3e-42e46ab79dc9/volumes" Nov 03 15:30:16.682196 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.682177 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" event={"ID":"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0","Type":"ContainerStarted","Data":"ea08c045d9d8ede95a2d22d904aa2b29343f1988fc0df9e2ff161910ab8b9e3b"} Nov 03 15:30:16.682269 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.682233 4290 scope.go:122] "RemoveContainer" containerID="281df00009c7ec7891f7b6802ea146709e183e40dd73acea0adf31ae59e1db7f" Nov 03 15:30:16.696410 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.696222 4290 scope.go:122] "RemoveContainer" containerID="281df00009c7ec7891f7b6802ea146709e183e40dd73acea0adf31ae59e1db7f" Nov 03 15:30:16.696511 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:16.696489 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281df00009c7ec7891f7b6802ea146709e183e40dd73acea0adf31ae59e1db7f\": container with ID starting with 281df00009c7ec7891f7b6802ea146709e183e40dd73acea0adf31ae59e1db7f not found: ID does not exist" containerID="281df00009c7ec7891f7b6802ea146709e183e40dd73acea0adf31ae59e1db7f" Nov 03 15:30:16.696553 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.696520 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281df00009c7ec7891f7b6802ea146709e183e40dd73acea0adf31ae59e1db7f"} err="failed to get container status \"281df00009c7ec7891f7b6802ea146709e183e40dd73acea0adf31ae59e1db7f\": rpc error: code = NotFound desc = could not find container \"281df00009c7ec7891f7b6802ea146709e183e40dd73acea0adf31ae59e1db7f\": container with ID starting with 281df00009c7ec7891f7b6802ea146709e183e40dd73acea0adf31ae59e1db7f not found: ID does not exist" Nov 03 15:30:16.791851 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.791814 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw"] Nov 03 15:30:16.935622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.935050 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2"] Nov 03 15:30:16.936554 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.936147 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:16.941657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.941269 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:16.941657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:16.941377 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:17.046154 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.046126 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" Nov 03 15:30:17.147318 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.147278 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee-kube-api-access-mz56b\" (UniqueName: \"kubernetes.io/projected/dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee-kube-api-access-mz56b\") pod \"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee\" (UID: \"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee\") " Nov 03 15:30:17.147664 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.147644 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee-maas-api-service-ca\") pod \"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee\" (UID: \"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee\") " Nov 03 15:30:17.148216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.148082 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee-maas-api-service-ca" pod "dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" (UID: "dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:17.148344 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.148323 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee-maas-api-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:17.149766 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.149734 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee-kube-api-access-mz56b" pod "dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" (UID: "dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee"). InnerVolumeSpecName "kube-api-access-mz56b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:17.250053 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.250020 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz56b\" (UniqueName: \"kubernetes.io/projected/dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee-kube-api-access-mz56b\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:17.328599 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.328511 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl"] Nov 03 15:30:17.330071 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.330045 4290 cpu_manager.go:414] "containerMap: removing container" podUID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" containerName="cleanup" Nov 03 15:30:17.330071 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.330070 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" containerName="cleanup" Nov 03 15:30:17.330239 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.330096 4290 cpu_manager.go:414] "containerMap: removing container" podUID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" containerName="cleanup" Nov 03 15:30:17.330239 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.330103 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" containerName="cleanup" Nov 03 15:30:17.330239 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.330126 4290 cpu_manager.go:414] "containerMap: removing container" podUID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" containerName="cleanup" Nov 03 15:30:17.330239 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.330134 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" containerName="cleanup" Nov 03 15:30:17.330378 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.330366 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" containerName="cleanup" Nov 03 15:30:17.330413 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.330395 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" containerName="cleanup" Nov 03 15:30:17.330413 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.330405 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" containerName="cleanup" Nov 03 15:30:17.334989 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.334966 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.407753 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.402776 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl"] Nov 03 15:30:17.453060 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.453029 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.453253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.453131 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.453253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.453181 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.453358 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.453293 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.453358 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.453343 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.453425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.453371 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.453425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.453412 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.453495 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.453484 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swkzb\" (UniqueName: \"kubernetes.io/projected/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-kube-api-access-swkzb\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.453555 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.453538 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.554799 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.554763 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.554998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.554857 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.554998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.554897 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.554998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.554942 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.554998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.554978 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.555227 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.555004 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.555227 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.555045 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.555227 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.555116 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swkzb\" (UniqueName: \"kubernetes.io/projected/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-kube-api-access-swkzb\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.555227 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.555168 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.555426 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.555234 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.555613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.555587 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.555865 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.555846 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.556145 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.556120 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.556522 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.556496 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.557935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.557907 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.559947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.559919 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.607294 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.605576 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl"] Nov 03 15:30:17.618536 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.615090 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.621613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.621567 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swkzb\" (UniqueName: \"kubernetes.io/projected/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-kube-api-access-swkzb\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.624217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.624157 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.629457 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.629431 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2"] Nov 03 15:30:17.638055 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.638025 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.661730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.652958 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:17.686205 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.683207 4290 scope.go:122] "RemoveContainer" containerID="651cf29f5f7d3e317f30d7f47de1c81a6fbba311c408678e9e303b3fc8cdf84b" Nov 03 15:30:17.722758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.722359 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" event={"ID":"dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee","Type":"ContainerDied","Data":"266590fd941ec10553b23fa7d6adb4cb3991e57f4155d9f53e5472b836cf090b"} Nov 03 15:30:17.722758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.722430 4290 scope.go:122] "RemoveContainer" containerID="e3afd4c895fe7bea91d7cd9c2eb7c42b04530b09ed1a55f994b813c90dfdaee4" Nov 03 15:30:17.722758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.722554 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw" Nov 03 15:30:17.726588 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.726566 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl"] Nov 03 15:30:17.736623 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.736598 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2"] Nov 03 15:30:17.792559 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.792537 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.794840 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.794812 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f213cf23-67be-4699-828e-23a0a966c12a-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.795016 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.794999 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.795218 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.795203 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfrll\" (UniqueName: \"kubernetes.io/projected/6d563b0f-d43c-4799-a59c-cf440af5506f-kube-api-access-zfrll\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.795318 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.795304 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d563b0f-d43c-4799-a59c-cf440af5506f-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.795425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.795413 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.795538 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.795526 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.795634 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.795622 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.795823 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.795806 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.795968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.795955 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.796079 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.796067 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.796187 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.796174 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.797038 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.797012 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.797202 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.797177 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.797345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.797225 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f213cf23-67be-4699-828e-23a0a966c12a-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.797345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.797249 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.797345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.797278 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f213cf23-67be-4699-828e-23a0a966c12a-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.797345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.797340 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlm89\" (UniqueName: \"kubernetes.io/projected/f213cf23-67be-4699-828e-23a0a966c12a-kube-api-access-tlm89\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.797564 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.793137 4290 generic.go:356] "Generic (PLEG): container finished" podID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" containerID="ea08c045d9d8ede95a2d22d904aa2b29343f1988fc0df9e2ff161910ab8b9e3b" exitCode=60 Nov 03 15:30:17.799135 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.793162 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" event={"ID":"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0","Type":"ContainerDied","Data":"ea08c045d9d8ede95a2d22d904aa2b29343f1988fc0df9e2ff161910ab8b9e3b"} Nov 03 15:30:17.799135 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.798170 4290 scope.go:122] "RemoveContainer" containerID="ea08c045d9d8ede95a2d22d904aa2b29343f1988fc0df9e2ff161910ab8b9e3b" Nov 03 15:30:17.799135 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:17.798410 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4_odh-ai-gateway-infra(d5ba8f6e-c38e-4072-b776-a5269ef2f7d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" podUID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" Nov 03 15:30:17.800990 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.800964 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" event={"ID":"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b","Type":"ContainerStarted","Data":"e00ee0890113793576bac8fc281a751b0b26d34cc21911850d4d011c06c1bc7e"} Nov 03 15:30:17.801080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.801014 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" event={"ID":"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b","Type":"ContainerStarted","Data":"25a5b8c723b6e9f4782b9b26d1aa239ce95730dd956139969cfa431f13adddd0"} Nov 03 15:30:17.811813 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.811793 4290 scope.go:122] "RemoveContainer" containerID="da6163157b70ca9c10ba278549c0c69c14c48d787391ace8902cfc930edd6003" Nov 03 15:30:17.919881 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.919843 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4"] Nov 03 15:30:17.920656 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.920622 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f213cf23-67be-4699-828e-23a0a966c12a-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.920795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.920723 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlm89\" (UniqueName: \"kubernetes.io/projected/f213cf23-67be-4699-828e-23a0a966c12a-kube-api-access-tlm89\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.920855 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.920838 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.920905 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.920877 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f213cf23-67be-4699-828e-23a0a966c12a-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.920971 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.920912 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.921047 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.921029 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfrll\" (UniqueName: \"kubernetes.io/projected/6d563b0f-d43c-4799-a59c-cf440af5506f-kube-api-access-zfrll\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.921402 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.921382 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d563b0f-d43c-4799-a59c-cf440af5506f-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.922398 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.922377 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d563b0f-d43c-4799-a59c-cf440af5506f-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.922842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.922820 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.923003 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.922984 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.923114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.923100 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.923220 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.923205 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.923340 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.923324 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.923471 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.923452 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.923595 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.923581 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.923824 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.923805 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.923962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.923945 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.924100 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.924082 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f213cf23-67be-4699-828e-23a0a966c12a-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.924205 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.924191 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.924670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.924650 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.925284 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.925023 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.925384 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.925322 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.925628 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.925603 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.927253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.926347 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.927253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.927185 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.927253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.927199 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f213cf23-67be-4699-828e-23a0a966c12a-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.928066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.927777 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.928066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.927894 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:17.931780 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.931762 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw"] Nov 03 15:30:17.954731 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.951080 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:17.954731 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.952741 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895330-6qrjw"] Nov 03 15:30:18.000724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.971279 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f213cf23-67be-4699-828e-23a0a966c12a-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:18.007918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.984804 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:18.007918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.985026 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:18.007918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:17.995064 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:18.018960 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.018751 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfrll\" (UniqueName: \"kubernetes.io/projected/6d563b0f-d43c-4799-a59c-cf440af5506f-kube-api-access-zfrll\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:18.022896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.022083 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:18.022896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.022787 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f213cf23-67be-4699-828e-23a0a966c12a-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:18.026427 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.026407 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlm89\" (UniqueName: \"kubernetes.io/projected/f213cf23-67be-4699-828e-23a0a966c12a-kube-api-access-tlm89\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:18.087144 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.067680 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj"] Nov 03 15:30:18.106378 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.106350 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.126577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.126541 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj"] Nov 03 15:30:18.127379 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.127343 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.127483 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.127462 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.127534 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.127515 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.127593 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.127578 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.127805 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.127781 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chwm5\" (UniqueName: \"kubernetes.io/projected/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-kube-api-access-chwm5\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.127898 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.127833 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.127898 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.127868 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.128023 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.127942 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.128023 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.127999 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.229746 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.229683 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.229896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.229859 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwm5\" (UniqueName: \"kubernetes.io/projected/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-kube-api-access-chwm5\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.229972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.229900 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.229972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.229929 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.230093 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.229985 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.230093 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.230035 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.230194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.230120 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.230194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.230119 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.230194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.230182 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.230345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.230227 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.231027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.230998 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.231167 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.231065 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.231288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.231263 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.231546 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.231522 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.267752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.247980 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:18.275021 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.274091 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.287037 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.286985 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.319730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.319672 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwm5\" (UniqueName: \"kubernetes.io/projected/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-kube-api-access-chwm5\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.320153 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.320046 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.388605 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.387502 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" podStartSLOduration=2.387479859 podStartE2EDuration="2.387479859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:18.378651773 +0000 UTC m=+1325.809609582" watchObservedRunningTime="2026-11-03 15:30:18.387479859 +0000 UTC m=+1325.818437666" Nov 03 15:30:18.406670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.405229 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs"] Nov 03 15:30:18.406798 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.406779 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:18.406839 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.406820 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" podUID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 15:30:18.410403 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.410374 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:18.410492 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.410468 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" Nov 03 15:30:18.469792 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.451983 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:18.483727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.476002 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:30:18.503027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.496984 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:18.503027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.497062 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:18.503027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.497103 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:18.506773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.506746 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl"] Nov 03 15:30:18.515748 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.510828 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs"] Nov 03 15:30:18.525667 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.525519 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2"] Nov 03 15:30:18.529788 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:30:18.528571 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d563b0f_d43c_4799_a59c_cf440af5506f.slice/crio-30ab7a983df15e387efa973c15a4cb24d84dab6627a71aaba24e4eca55b999bf WatchSource:0}: Error finding container 30ab7a983df15e387efa973c15a4cb24d84dab6627a71aaba24e4eca55b999bf: Status 404 returned error can't find the container with id 30ab7a983df15e387efa973c15a4cb24d84dab6627a71aaba24e4eca55b999bf Nov 03 15:30:18.531849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.531814 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:18.531953 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.531891 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:18.531953 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.531936 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:18.579177 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.540244 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d955c5f2-5757-4b5e-9667-9e3db67795bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ab54ac5-lbrfs\" (UID: \"d955c5f2-5757-4b5e-9667-9e3db67795bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" Nov 03 15:30:18.579177 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.540393 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d955c5f2-5757-4b5e-9667-9e3db67795bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ab54ac5-lbrfs\" (UID: \"d955c5f2-5757-4b5e-9667-9e3db67795bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" Nov 03 15:30:18.579177 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.540507 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m959\" (UniqueName: \"kubernetes.io/projected/d955c5f2-5757-4b5e-9667-9e3db67795bf-kube-api-access-5m959\") pod \"maas-api-revoke-sub-8ab54ac5-lbrfs\" (UID: \"d955c5f2-5757-4b5e-9667-9e3db67795bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" Nov 03 15:30:18.676995 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.663686 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d955c5f2-5757-4b5e-9667-9e3db67795bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ab54ac5-lbrfs\" (UID: \"d955c5f2-5757-4b5e-9667-9e3db67795bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" Nov 03 15:30:18.676995 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.663861 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d955c5f2-5757-4b5e-9667-9e3db67795bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ab54ac5-lbrfs\" (UID: \"d955c5f2-5757-4b5e-9667-9e3db67795bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" Nov 03 15:30:18.676995 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.663983 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m959\" (UniqueName: \"kubernetes.io/projected/d955c5f2-5757-4b5e-9667-9e3db67795bf-kube-api-access-5m959\") pod \"maas-api-revoke-sub-8ab54ac5-lbrfs\" (UID: \"d955c5f2-5757-4b5e-9667-9e3db67795bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" Nov 03 15:30:18.676995 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.665205 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d955c5f2-5757-4b5e-9667-9e3db67795bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ab54ac5-lbrfs\" (UID: \"d955c5f2-5757-4b5e-9667-9e3db67795bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" Nov 03 15:30:18.689895 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.689497 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d955c5f2-5757-4b5e-9667-9e3db67795bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ab54ac5-lbrfs\" (UID: \"d955c5f2-5757-4b5e-9667-9e3db67795bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" Nov 03 15:30:18.698969 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.698939 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee" path="/var/lib/kubelet/pods/dfc74c57-6ad7-464b-a9fd-b1eebeaf1dee/volumes" Nov 03 15:30:18.770581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.745359 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m959\" (UniqueName: \"kubernetes.io/projected/d955c5f2-5757-4b5e-9667-9e3db67795bf-kube-api-access-5m959\") pod \"maas-api-revoke-sub-8ab54ac5-lbrfs\" (UID: \"d955c5f2-5757-4b5e-9667-9e3db67795bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" Nov 03 15:30:18.802080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.795221 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:18.802080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.795311 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:18.802080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.795354 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:18.802080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.799518 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl"] Nov 03 15:30:18.818142 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.818047 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" Nov 03 15:30:18.934947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:18.901891 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" event={"ID":"6d563b0f-d43c-4799-a59c-cf440af5506f","Type":"ContainerStarted","Data":"30ab7a983df15e387efa973c15a4cb24d84dab6627a71aaba24e4eca55b999bf"} Nov 03 15:30:19.007141 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.000561 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" event={"ID":"f213cf23-67be-4699-828e-23a0a966c12a","Type":"ContainerStarted","Data":"4023dfd9c2aa3cb673305ff17e8caddb9d34cff53cc175290b08a98838ccfb2d"} Nov 03 15:30:19.097146 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.084242 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj"] Nov 03 15:30:19.199791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.192495 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" event={"ID":"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f","Type":"ContainerStarted","Data":"daf79fbd9b9595eed6dda3288851767c79a95f9798d46e7c8ef0991d7560b89c"} Nov 03 15:30:19.199791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.192563 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" event={"ID":"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f","Type":"ContainerStarted","Data":"d087da7adc371bbb41fa66d50ef5c0d61f563d3da2de74d4a0122c56fc25472f"} Nov 03 15:30:19.199791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.197996 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" event={"ID":"8bde81d0-b4d2-4393-b052-e550d9241d2e","Type":"ContainerStarted","Data":"1832e80f6e9d80a89bc256faac8c88b21348ca9a17de7f9c4abf87b59ff36658"} Nov 03 15:30:19.199791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.198445 4290 scope.go:122] "RemoveContainer" containerID="1832e80f6e9d80a89bc256faac8c88b21348ca9a17de7f9c4abf87b59ff36658" Nov 03 15:30:19.199791 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:19.198821 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz_odh-ai-gateway-infra(8bde81d0-b4d2-4393-b052-e550d9241d2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" podUID="8bde81d0-b4d2-4393-b052-e550d9241d2e" Nov 03 15:30:19.222052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.203082 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:19.222052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.203164 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:19.222052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.203205 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:19.474729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.466300 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:19.474729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.466359 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" podUID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 15:30:19.478725 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.477590 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podStartSLOduration=2.477570503 podStartE2EDuration="2.477570503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:19.47589248 +0000 UTC m=+1326.906850287" watchObservedRunningTime="2026-11-03 15:30:19.477570503 +0000 UTC m=+1326.908528311" Nov 03 15:30:19.671723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.654158 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:19.705592 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.705561 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:19.705901 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.705862 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 15:30:19.818118 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:19.818089 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs"] Nov 03 15:30:20.033298 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.033276 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" Nov 03 15:30:20.210377 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.210346 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ba8f6e-c38e-4072-b776-a5269ef2f7d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ba8f6e-c38e-4072-b776-a5269ef2f7d0-maas-api-service-ca\") pod \"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0\" (UID: \"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0\") " Nov 03 15:30:20.212067 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.211802 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ba8f6e-c38e-4072-b776-a5269ef2f7d0-kube-api-access-hj4mq\" (UniqueName: \"kubernetes.io/projected/d5ba8f6e-c38e-4072-b776-a5269ef2f7d0-kube-api-access-hj4mq\") pod \"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0\" (UID: \"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0\") " Nov 03 15:30:20.212396 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.212374 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ba8f6e-c38e-4072-b776-a5269ef2f7d0-maas-api-service-ca" pod "d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" (UID: "d5ba8f6e-c38e-4072-b776-a5269ef2f7d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:20.212804 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.212786 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ba8f6e-c38e-4072-b776-a5269ef2f7d0-maas-api-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:20.215063 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.215041 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ba8f6e-c38e-4072-b776-a5269ef2f7d0-kube-api-access-hj4mq" pod "d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" (UID: "d5ba8f6e-c38e-4072-b776-a5269ef2f7d0"). InnerVolumeSpecName "kube-api-access-hj4mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:20.215918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.215649 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" event={"ID":"6d563b0f-d43c-4799-a59c-cf440af5506f","Type":"ContainerStarted","Data":"785ff0e19bcba4146ba263084590b2b2e603fa23bef4dca752506cb4a1e6e396"} Nov 03 15:30:20.217430 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.217400 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" event={"ID":"d955c5f2-5757-4b5e-9667-9e3db67795bf","Type":"ContainerStarted","Data":"05a3795cd01774e4b9580bf4996e420550c9960c353b8e58e859ccf06cb34bf4"} Nov 03 15:30:20.219864 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.219835 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" event={"ID":"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8","Type":"ContainerStarted","Data":"48472a61f95b896b487d15500b7ef9923da33dee9121d930c06e01579a83fbe9"} Nov 03 15:30:20.219954 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.219892 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" event={"ID":"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8","Type":"ContainerStarted","Data":"28df500608325cf572957a867791b001f28c01ca7dc47bb949d8d5c33151634c"} Nov 03 15:30:20.222404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.222379 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" Nov 03 15:30:20.222404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.222386 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4" event={"ID":"d5ba8f6e-c38e-4072-b776-a5269ef2f7d0","Type":"ContainerDied","Data":"7c727f38bf16fd4022787897dbf06b4e1a67bf6e2d097ce3a33ea5f0d21fbff3"} Nov 03 15:30:20.222552 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.222435 4290 scope.go:122] "RemoveContainer" containerID="ea08c045d9d8ede95a2d22d904aa2b29343f1988fc0df9e2ff161910ab8b9e3b" Nov 03 15:30:20.227074 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.225154 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" event={"ID":"f213cf23-67be-4699-828e-23a0a966c12a","Type":"ContainerStarted","Data":"946748a29710d727cba118f7cda64bdc989bde662f7da4e72167a1b11fe0000a"} Nov 03 15:30:20.276754 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.273355 4290 generic.go:356] "Generic (PLEG): container finished" podID="8bde81d0-b4d2-4393-b052-e550d9241d2e" containerID="1832e80f6e9d80a89bc256faac8c88b21348ca9a17de7f9c4abf87b59ff36658" exitCode=60 Nov 03 15:30:20.276754 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.274845 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" event={"ID":"8bde81d0-b4d2-4393-b052-e550d9241d2e","Type":"ContainerDied","Data":"1832e80f6e9d80a89bc256faac8c88b21348ca9a17de7f9c4abf87b59ff36658"} Nov 03 15:30:20.276754 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.275406 4290 scope.go:122] "RemoveContainer" containerID="1832e80f6e9d80a89bc256faac8c88b21348ca9a17de7f9c4abf87b59ff36658" Nov 03 15:30:20.276754 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:20.275905 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz_odh-ai-gateway-infra(8bde81d0-b4d2-4393-b052-e550d9241d2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" podUID="8bde81d0-b4d2-4393-b052-e550d9241d2e" Nov 03 15:30:20.286913 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.286469 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:20.289711 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.289655 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:20.289847 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.289784 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 15:30:20.317584 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.317553 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj4mq\" (UniqueName: \"kubernetes.io/projected/d5ba8f6e-c38e-4072-b776-a5269ef2f7d0-kube-api-access-hj4mq\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:20.323957 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.323924 4290 scope.go:122] "RemoveContainer" containerID="651cf29f5f7d3e317f30d7f47de1c81a6fbba311c408678e9e303b3fc8cdf84b" Nov 03 15:30:20.413859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.413526 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:20.413859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.413605 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" podUID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 15:30:20.458383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.458356 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:20.459592 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.459557 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:30:20.463965 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.463933 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 15:30:20.510724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.510310 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" podStartSLOduration=3.510291698 podStartE2EDuration="3.510291698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:20.390180952 +0000 UTC m=+1327.821138759" watchObservedRunningTime="2026-11-03 15:30:20.510291698 +0000 UTC m=+1327.941249504" Nov 03 15:30:20.533796 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.518572 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4"] Nov 03 15:30:20.609337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.603601 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895330-x6jb4"] Nov 03 15:30:20.770334 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.770297 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:20.770615 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:20.770586 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 15:30:21.081539 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.067115 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:21.081539 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.067197 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 15:30:21.266808 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.266261 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" path="/var/lib/kubelet/pods/d5ba8f6e-c38e-4072-b776-a5269ef2f7d0/volumes" Nov 03 15:30:21.266808 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.266800 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz"] Nov 03 15:30:21.267357 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.266875 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:21.346786 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.346440 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podStartSLOduration=4.346416423 podStartE2EDuration="4.346416423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:20.670875485 +0000 UTC m=+1328.101833294" watchObservedRunningTime="2026-11-03 15:30:21.346416423 +0000 UTC m=+1328.777374230" Nov 03 15:30:21.528745 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.382603 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 15:30:21.528745 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.412612 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:21.528745 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.423536 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:21.528745 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.423596 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" podUID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 15:30:21.528745 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.457021 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 15:30:21.528745 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.500622 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podStartSLOduration=4.500601275 podStartE2EDuration="4.500601275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:20.930027671 +0000 UTC m=+1328.360985478" watchObservedRunningTime="2026-11-03 15:30:21.500601275 +0000 UTC m=+1328.931559082" Nov 03 15:30:21.562767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.529914 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:30:21.562767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.529985 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 15:30:21.605833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.564076 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:21.605833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.580499 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:21.605833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.580552 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 15:30:21.702935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.702897 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:21.703223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.703195 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 15:30:21.708938 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.707772 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" event={"ID":"d955c5f2-5757-4b5e-9667-9e3db67795bf","Type":"ContainerStarted","Data":"a26c882371d548259ace46eb2d6163e80a8932c4c4f62f13f27c29b9d8967225"} Nov 03 15:30:21.842407 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.842359 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:21.843033 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.843004 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Nov 03 15:30:21.879792 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:21.861326 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" podStartSLOduration=3.8613029340000002 podStartE2EDuration="3.861302934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:21.859108667 +0000 UTC m=+1329.290066477" watchObservedRunningTime="2026-11-03 15:30:21.861302934 +0000 UTC m=+1329.292260765" Nov 03 15:30:22.052837 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.052798 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:22.053035 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.052862 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 15:30:22.349793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.337340 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:22.349793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.337410 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 15:30:22.431373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.409069 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 15:30:22.431373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.413606 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:22.431373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.413657 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" podUID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 15:30:22.431373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.413756 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Nov 03 15:30:22.431373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.414041 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:30:22.461761 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.435824 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:22.461761 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.436259 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:30:22.500118 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.486302 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:30:22.500118 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.486361 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 15:30:22.629742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.627051 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:22.629742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.627427 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 15:30:22.629742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.627771 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:30:22.737018 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.736969 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:22.737277 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:22.737253 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 15:30:23.116397 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.116352 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:23.123321 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.123269 4290 generic.go:356] "Generic (PLEG): container finished" podID="d955c5f2-5757-4b5e-9667-9e3db67795bf" containerID="a26c882371d548259ace46eb2d6163e80a8932c4c4f62f13f27c29b9d8967225" exitCode=0 Nov 03 15:30:23.141336 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.138541 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" event={"ID":"d955c5f2-5757-4b5e-9667-9e3db67795bf","Type":"ContainerDied","Data":"a26c882371d548259ace46eb2d6163e80a8932c4c4f62f13f27c29b9d8967225"} Nov 03 15:30:23.141336 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.138632 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:30:23.193848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.174398 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" Nov 03 15:30:23.193848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.185470 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:23.193848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.185525 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 15:30:23.351989 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.224776 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bde81d0-b4d2-4393-b052-e550d9241d2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bde81d0-b4d2-4393-b052-e550d9241d2e-maas-api-service-ca\") pod \"8bde81d0-b4d2-4393-b052-e550d9241d2e\" (UID: \"8bde81d0-b4d2-4393-b052-e550d9241d2e\") " Nov 03 15:30:23.374225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.374192 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bde81d0-b4d2-4393-b052-e550d9241d2e-kube-api-access-8rxqf\" (UniqueName: \"kubernetes.io/projected/8bde81d0-b4d2-4393-b052-e550d9241d2e-kube-api-access-8rxqf\") pod \"8bde81d0-b4d2-4393-b052-e550d9241d2e\" (UID: \"8bde81d0-b4d2-4393-b052-e550d9241d2e\") " Nov 03 15:30:23.392867 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.227277 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bde81d0-b4d2-4393-b052-e550d9241d2e-maas-api-service-ca" pod "8bde81d0-b4d2-4393-b052-e550d9241d2e" (UID: "8bde81d0-b4d2-4393-b052-e550d9241d2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:23.392867 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.359811 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:23.392867 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.388928 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 15:30:23.430795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.415915 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Nov 03 15:30:23.447080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.437157 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:23.447080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.437210 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" podUID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 15:30:23.487487 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.487443 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:23.507723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.488328 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:23.513411 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.488628 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:30:23.513825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.513658 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 15:30:23.533842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.525109 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bde81d0-b4d2-4393-b052-e550d9241d2e-maas-api-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:23.703745 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.694368 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bde81d0-b4d2-4393-b052-e550d9241d2e-kube-api-access-8rxqf" pod "8bde81d0-b4d2-4393-b052-e550d9241d2e" (UID: "8bde81d0-b4d2-4393-b052-e550d9241d2e"). InnerVolumeSpecName "kube-api-access-8rxqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:23.843734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.802866 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:23.843734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:23.831604 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rxqf\" (UniqueName: \"kubernetes.io/projected/8bde81d0-b4d2-4393-b052-e550d9241d2e-kube-api-access-8rxqf\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:24.167728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.151378 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 15:30:24.233771 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.220300 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:24.233771 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.220364 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 15:30:24.314729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.310786 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 15:30:24.534164 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.433893 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:24.534164 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.519284 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:24.534164 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.519370 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 15:30:24.566805 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.566760 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 15:30:24.737294 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.737249 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:30:24.737601 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.737559 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 15:30:24.775027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.772850 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:24.775027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.773720 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:30:24.884566 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.859832 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:24.884566 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.859914 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 15:30:24.966737 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.954417 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:24.967073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:24.967034 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" podUID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 15:30:25.008518 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.008472 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 15:30:25.041152 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.041111 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:25.067735 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.043452 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Nov 03 15:30:25.178777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.172391 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:25.178777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.172521 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:25.187173 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.186506 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 15:30:25.356613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.356560 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:30:25.388755 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.376003 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:25.388755 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.378108 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:25.388755 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.378151 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 15:30:25.422297 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.416415 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:25.422297 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.416477 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 15:30:25.437791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.426731 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:25.437791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.426783 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" podUID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 15:30:25.510085 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.510036 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:30:25.554742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.548080 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 15:30:25.718686 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.614980 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 15:30:25.729026 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.615028 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:25.867165 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.867126 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:25.867633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.867602 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:25.879767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.879715 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 15:30:25.981787 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.949375 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:30:25.985894 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:25.985866 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:30:26.037115 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.021945 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:26.072508 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.072483 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:30:26.179395 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.116350 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" Nov 03 15:30:26.240761 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.179342 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz" event={"ID":"8bde81d0-b4d2-4393-b052-e550d9241d2e","Type":"ContainerDied","Data":"6adb86e0da93de727802e58e7fb3aad4aacf82cf9259cf1703dd60b4ceab44c7"} Nov 03 15:30:26.241073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.241054 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:30:26.241230 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.241218 4290 scope.go:122] "RemoveContainer" containerID="1832e80f6e9d80a89bc256faac8c88b21348ca9a17de7f9c4abf87b59ff36658" Nov 03 15:30:26.266900 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.145790 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:26.267258 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.267229 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 15:30:26.335125 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.312120 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:26.335125 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.312173 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 15:30:26.538160 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.433554 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:26.538160 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.433612 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" podUID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 15:30:26.538160 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.449935 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 15:30:26.538160 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.450377 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:26.538160 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.477002 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 15:30:26.538160 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.487912 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:30:26.638732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.626540 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:30:26.638732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.626595 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 15:30:26.782752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.737870 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:26.782752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.738030 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:26.782752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.738060 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 15:30:26.782752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.741295 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:26.782752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.741632 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 15:30:26.994802 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:26.868800 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 15:30:27.078730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.053940 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:27.085638 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.085598 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 15:30:27.142730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.122978 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 15:30:27.142730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.124202 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:27.209738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.191785 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded" Nov 03 15:30:27.288890 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.288802 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:27.289136 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.289107 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 15:30:27.375308 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.368539 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded" Nov 03 15:30:27.444728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.418913 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:27.445574 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.445518 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" podUID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 15:30:27.469762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.450754 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:27.469762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.468620 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:27.499521 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.489783 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:27.499521 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.495679 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:30:27.499521 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.495763 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 15:30:27.622794 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.606904 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd"] Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656235 4290 cpu_manager.go:414] "containerMap: removing container" podUID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" containerName="cleanup" Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656266 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" containerName="cleanup" Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656291 4290 cpu_manager.go:414] "containerMap: removing container" podUID="8bde81d0-b4d2-4393-b052-e550d9241d2e" containerName="cleanup" Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656299 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bde81d0-b4d2-4393-b052-e550d9241d2e" containerName="cleanup" Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656319 4290 cpu_manager.go:414] "containerMap: removing container" podUID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" containerName="cleanup" Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656327 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" containerName="cleanup" Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656348 4290 cpu_manager.go:414] "containerMap: removing container" podUID="8bde81d0-b4d2-4393-b052-e550d9241d2e" containerName="cleanup" Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656355 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bde81d0-b4d2-4393-b052-e550d9241d2e" containerName="cleanup" Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656387 4290 cpu_manager.go:414] "containerMap: removing container" podUID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" containerName="cleanup" Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656394 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" containerName="cleanup" Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656557 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bde81d0-b4d2-4393-b052-e550d9241d2e" containerName="cleanup" Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656593 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" containerName="cleanup" Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656611 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" containerName="cleanup" Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656622 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bde81d0-b4d2-4393-b052-e550d9241d2e" containerName="cleanup" Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656633 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ba8f6e-c38e-4072-b776-a5269ef2f7d0" containerName="cleanup" Nov 03 15:30:27.657728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.656644 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bde81d0-b4d2-4393-b052-e550d9241d2e" containerName="cleanup" Nov 03 15:30:27.772742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.705751 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:27.772742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.705845 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 15:30:27.772742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.766026 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd"] Nov 03 15:30:27.772742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.767089 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" Nov 03 15:30:27.772742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.768573 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:27.772742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.768669 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:27.773383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.773350 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 15:30:27.828727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.825273 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-47b989b3\"" Nov 03 15:30:27.863789 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.863753 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-47b989b3-dockercfg-4r7f9\"" Nov 03 15:30:27.893776 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.885186 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:27.998196 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.983531 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd8c668-f317-4c36-b64a-e4dab6a9cfe0-plugins-config-volume\") pod \"payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd\" (UID: \"1bd8c668-f317-4c36-b64a-e4dab6a9cfe0\") " pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" Nov 03 15:30:27.998196 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:27.983684 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxf5x\" (UniqueName: \"kubernetes.io/projected/1bd8c668-f317-4c36-b64a-e4dab6a9cfe0-kube-api-access-wxf5x\") pod \"payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd\" (UID: \"1bd8c668-f317-4c36-b64a-e4dab6a9cfe0\") " pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" Nov 03 15:30:28.046880 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.046837 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:28.047069 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.046903 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 15:30:28.099237 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.099199 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd8c668-f317-4c36-b64a-e4dab6a9cfe0-plugins-config-volume\") pod \"payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd\" (UID: \"1bd8c668-f317-4c36-b64a-e4dab6a9cfe0\") " pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" Nov 03 15:30:28.102728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.099552 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxf5x\" (UniqueName: \"kubernetes.io/projected/1bd8c668-f317-4c36-b64a-e4dab6a9cfe0-kube-api-access-wxf5x\") pod \"payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd\" (UID: \"1bd8c668-f317-4c36-b64a-e4dab6a9cfe0\") " pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" Nov 03 15:30:28.113885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.110412 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd8c668-f317-4c36-b64a-e4dab6a9cfe0-plugins-config-volume\") pod \"payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd\" (UID: \"1bd8c668-f317-4c36-b64a-e4dab6a9cfe0\") " pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" Nov 03 15:30:28.128395 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.128365 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:30:28.138732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.129805 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:28.139236 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.134785 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:28.164772 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.139628 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7"] Nov 03 15:30:28.166176 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.166145 4290 cpu_manager.go:414] "containerMap: removing container" podUID="8bde81d0-b4d2-4393-b052-e550d9241d2e" containerName="cleanup" Nov 03 15:30:28.234544 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.234510 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bde81d0-b4d2-4393-b052-e550d9241d2e" containerName="cleanup" Nov 03 15:30:28.284137 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.274174 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:28.289230 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.289193 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 15:30:28.343197 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.334369 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxf5x\" (UniqueName: \"kubernetes.io/projected/1bd8c668-f317-4c36-b64a-e4dab6a9cfe0-kube-api-access-wxf5x\") pod \"payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd\" (UID: \"1bd8c668-f317-4c36-b64a-e4dab6a9cfe0\") " pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" Nov 03 15:30:28.343197 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.334639 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" Nov 03 15:30:28.381091 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.379984 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:30:28.381091 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.380100 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" Nov 03 15:30:28.458741 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.458682 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:30:28.476778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.460773 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 15:30:28.477972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.477936 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 15:30:28.513754 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.497273 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:28.566850 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.566808 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce976803-7a19-4e58-a3bc-e1132338b811-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7\" (UID: \"ce976803-7a19-4e58-a3bc-e1132338b811\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" Nov 03 15:30:28.593860 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.569857 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgckx\" (UniqueName: \"kubernetes.io/projected/ce976803-7a19-4e58-a3bc-e1132338b811-kube-api-access-zgckx\") pod \"payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7\" (UID: \"ce976803-7a19-4e58-a3bc-e1132338b811\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" Nov 03 15:30:28.689902 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.687861 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:28.729636 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.691773 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 15:30:28.786728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.773597 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 15:30:28.840738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.839509 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Nov 03 15:30:28.888287 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.888245 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:28.896356 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.891234 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce976803-7a19-4e58-a3bc-e1132338b811-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7\" (UID: \"ce976803-7a19-4e58-a3bc-e1132338b811\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" Nov 03 15:30:28.896356 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.891324 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgckx\" (UniqueName: \"kubernetes.io/projected/ce976803-7a19-4e58-a3bc-e1132338b811-kube-api-access-zgckx\") pod \"payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7\" (UID: \"ce976803-7a19-4e58-a3bc-e1132338b811\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" Nov 03 15:30:28.910035 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:28.898559 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce976803-7a19-4e58-a3bc-e1132338b811-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7\" (UID: \"ce976803-7a19-4e58-a3bc-e1132338b811\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" Nov 03 15:30:29.411937 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.411902 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgckx\" (UniqueName: \"kubernetes.io/projected/ce976803-7a19-4e58-a3bc-e1132338b811-kube-api-access-zgckx\") pod \"payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7\" (UID: \"ce976803-7a19-4e58-a3bc-e1132338b811\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" Nov 03 15:30:29.431112 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.431067 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:29.435833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.435795 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 15:30:29.461734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.438227 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:29.461734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.438305 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 15:30:29.466109 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.466076 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:29.492612 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.471142 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 15:30:29.492612 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.479042 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:30:29.492612 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.479118 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 15:30:29.676891 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.676773 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:29.706062 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.706013 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 15:30:29.841144 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.841091 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:29.842171 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.778112 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:29.856011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.842393 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 15:30:30.105790 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.844128 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" Nov 03 15:30:30.105790 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.844662 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 15:30:30.106023 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:29.997626 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 15:30:30.124494 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.050581 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:30.124494 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.107002 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 15:30:30.186294 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:30.186245 4290 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.516s" Nov 03 15:30:30.457241 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.457201 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7"] Nov 03 15:30:30.457863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.457842 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:30.559403 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.546832 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" Nov 03 15:30:30.750725 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.750659 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" event={"ID":"d955c5f2-5757-4b5e-9667-9e3db67795bf","Type":"ContainerDied","Data":"05a3795cd01774e4b9580bf4996e420550c9960c353b8e58e859ccf06cb34bf4"} Nov 03 15:30:30.883877 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.855331 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05a3795cd01774e4b9580bf4996e420550c9960c353b8e58e859ccf06cb34bf4" Nov 03 15:30:30.924821 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.421012 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:30.942943 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.942877 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 15:30:30.960098 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.960051 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:30.960389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.960353 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 15:30:30.960629 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.960608 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Nov 03 15:30:30.960995 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.960973 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 15:30:31.197793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.331381 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d955c5f2-5757-4b5e-9667-9e3db67795bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d955c5f2-5757-4b5e-9667-9e3db67795bf-maas-api-xks-ca\") pod \"d955c5f2-5757-4b5e-9667-9e3db67795bf\" (UID: \"d955c5f2-5757-4b5e-9667-9e3db67795bf\") " Nov 03 15:30:31.197793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.978965 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 15:30:31.197793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:30.979152 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Nov 03 15:30:31.197793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.030946 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d955c5f2-5757-4b5e-9667-9e3db67795bf-kube-api-access-5m959\" (UniqueName: \"kubernetes.io/projected/d955c5f2-5757-4b5e-9667-9e3db67795bf-kube-api-access-5m959\") pod \"d955c5f2-5757-4b5e-9667-9e3db67795bf\" (UID: \"d955c5f2-5757-4b5e-9667-9e3db67795bf\") " Nov 03 15:30:31.197793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.031253 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d955c5f2-5757-4b5e-9667-9e3db67795bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d955c5f2-5757-4b5e-9667-9e3db67795bf-maas-api-openshift-ca\") pod \"d955c5f2-5757-4b5e-9667-9e3db67795bf\" (UID: \"d955c5f2-5757-4b5e-9667-9e3db67795bf\") " Nov 03 15:30:31.197793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.037318 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:30:31.197793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.044833 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:31.197793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.140079 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:31.197793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.140384 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 15:30:31.462614 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.461469 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qzlq" podUID="208146e8-fd8f-4d34-8e96-61c16cd5d4bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 15:30:31.514750 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.474738 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:31.514750 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.474795 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 15:30:31.514750 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.475936 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:31.514750 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.475974 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 15:30:31.525894 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.516584 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d955c5f2-5757-4b5e-9667-9e3db67795bf-maas-api-openshift-ca" pod "d955c5f2-5757-4b5e-9667-9e3db67795bf" (UID: "d955c5f2-5757-4b5e-9667-9e3db67795bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:31.578581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.575538 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d955c5f2-5757-4b5e-9667-9e3db67795bf-maas-api-xks-ca" pod "d955c5f2-5757-4b5e-9667-9e3db67795bf" (UID: "d955c5f2-5757-4b5e-9667-9e3db67795bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:30:31.578581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.576081 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 15:30:31.578581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.577936 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:30:31.578581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.577997 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 15:30:31.614864 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.614662 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:30:31.614864 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.614734 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:30:31.614864 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.614755 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:31.626716 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.626669 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": context deadline exceeded" Nov 03 15:30:31.627026 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.627003 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:30:31.639433 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.634799 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:30:31.933727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.906397 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:30:31.933727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.906478 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:30:31.935832 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.935675 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d955c5f2-5757-4b5e-9667-9e3db67795bf-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:31.936045 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.936025 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d955c5f2-5757-4b5e-9667-9e3db67795bf-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:31.945369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.945343 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:31.945985 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.945952 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 15:30:31.968558 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.968515 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:32.004770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:31.998894 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d955c5f2-5757-4b5e-9667-9e3db67795bf-kube-api-access-5m959" pod "d955c5f2-5757-4b5e-9667-9e3db67795bf" (UID: "d955c5f2-5757-4b5e-9667-9e3db67795bf"). InnerVolumeSpecName "kube-api-access-5m959". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:32.075773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.068118 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 15:30:32.102756 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.087219 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:32.102756 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.087276 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 15:30:32.102756 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.091559 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:30:32.112389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.103468 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:30:32.134516 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.134478 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m959\" (UniqueName: \"kubernetes.io/projected/d955c5f2-5757-4b5e-9667-9e3db67795bf-kube-api-access-5m959\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:32.201676 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.201646 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:30:32.205893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.205858 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:32.390043 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.325261 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" event={"ID":"1bd8c668-f317-4c36-b64a-e4dab6a9cfe0","Type":"ContainerStarted","Data":"3a0fb77b487a1af9477ed3a38682bbe6e602946f7e57df17335a2938b9c4f22a"} Nov 03 15:30:32.390043 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.325331 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz"] Nov 03 15:30:32.390043 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.326235 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs" Nov 03 15:30:32.390043 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.363786 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:32.390043 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.363850 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 15:30:32.428870 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.428837 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:32.432581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.432547 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 15:30:32.434906 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.434878 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895330-df7nz"] Nov 03 15:30:32.489720 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.489661 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:30:32.491584 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.491551 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 15:30:32.655452 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.655012 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:32.666939 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.666907 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:32.697184 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:32.697120 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 15:30:33.090770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:33.090734 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bde81d0-b4d2-4393-b052-e550d9241d2e" path="/var/lib/kubelet/pods/8bde81d0-b4d2-4393-b052-e550d9241d2e/volumes" Nov 03 15:30:33.590729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:33.566074 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:33.723960 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:33.723895 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 15:30:33.751779 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:33.727037 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:30:33.897519 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:33.897473 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:33.971093 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:33.971052 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:34.090987 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.090883 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 15:30:34.206725 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.206587 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:30:34.223860 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.208769 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:34.317767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.313639 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:34.317767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.313749 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" event={"ID":"ce976803-7a19-4e58-a3bc-e1132338b811","Type":"ContainerStarted","Data":"08dcdad6c3a31898d22718c7278c1c12b5ff248d51f52d9313c97605d8494c7b"} Nov 03 15:30:34.317767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.313823 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5"] Nov 03 15:30:34.317767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.313879 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz"] Nov 03 15:30:34.317767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.314145 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" podUID="3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" containerName="istio-proxy" containerID="cri-o://329b780fceb9d3e4217873ee3192ff75c2a45244adc17de72892b323f0f759ac" gracePeriod=30 Nov 03 15:30:34.333068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.319452 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" podUID="24535df2-4641-492e-aa5f-9bcdf1273d64" containerName="istio-proxy" containerID="cri-o://1c72926dda89df4304626630fd477598f9183f2d096bc266174cab3dc676dc71" gracePeriod=30 Nov 03 15:30:34.519320 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.519279 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" event={"ID":"1bd8c668-f317-4c36-b64a-e4dab6a9cfe0","Type":"ContainerStarted","Data":"62941ec3e17a96a6fcd81573d139d92d712669478163c630888d35d9d090e8c7"} Nov 03 15:30:34.520512 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.520477 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" Nov 03 15:30:34.772790 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.725085 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:34.772790 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.725525 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:34.810897 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.810839 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 15:30:34.829529 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.829463 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:34.831609 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.829848 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 15:30:34.853343 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.853310 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:30:34.853658 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.853594 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:34.884533 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.884496 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:30:34.884928 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.884894 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:34.897543 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.897500 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:30:34.897859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:34.897811 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.369958 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.369873 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd"] Nov 03 15:30:35.370573 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.370549 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:30:35.401727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.398158 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.460049 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.431819 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:35.463967 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.460519 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:30:35.472825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.437587 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.574728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.545520 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:30:35.667830 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.667787 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lcdb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:30:35.668090 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.668062 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" podUID="24535df2-4641-492e-aa5f-9bcdf1273d64" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:30:35.742734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.730094 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.761732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.731428 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.805734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.786371 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:30:35.816731 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.811484 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.927524 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.748142 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.944873 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.834854 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 15:30:36.007728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.834881 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:36.007728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.915676 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:30:36.007728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.005297 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:30:36.007728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.005341 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:30:36.007728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.945144 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:30:36.007728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.006027 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:36.050952 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.033623 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:30:36.050952 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:35.960490 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:30:36.050952 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.033747 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:36.167280 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.167245 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:30:36.193225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.193092 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:36.434640 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.388218 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:36.490723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.471200 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Nov 03 15:30:36.561730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.527465 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:30:36.561730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.527777 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:36.561730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.548589 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 15:30:36.750109 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.750001 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:30:36.761405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.761319 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:36.807260 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.807195 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Nov 03 15:30:36.810021 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.809985 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:36.892759 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.892726 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nm4sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:30:36.893066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.893020 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" podUID="3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:30:36.910927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.910870 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded" Nov 03 15:30:36.911255 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.911232 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 15:30:36.951036 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.950967 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 15:30:36.984241 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:36.984182 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.020919 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:37.020820 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.116634 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:37.116602 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:30:37.238908 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:37.238857 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:30:37.239218 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:37.239168 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.575690 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:37.575632 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 15:30:37.637119 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:37.637074 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded" Nov 03 15:30:37.731728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:37.575029 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.790725 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:37.766137 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.930727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:37.797732 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.930727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:37.823743 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:38.062726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:38.056065 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 15:30:38.082327 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:38.049076 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:38.152724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:38.099004 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 15:30:38.263776 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:38.105904 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 15:30:38.263776 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:38.120083 4290 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.003s" Nov 03 15:30:38.263776 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:38.263214 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7"] Nov 03 15:30:38.325729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:38.290486 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:30:38.325729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:38.290548 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:30:38.357758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:38.347406 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" event={"ID":"ce976803-7a19-4e58-a3bc-e1132338b811","Type":"ContainerStarted","Data":"9106e69183de05b32b43795db0c4d813ee855320af884a1fe0bd2e8a1fedd686"} Nov 03 15:30:38.357758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:38.348637 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" Nov 03 15:30:38.449919 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:38.449885 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" Nov 03 15:30:38.626784 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:38.578279 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 15:30:38.784950 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:38.784906 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:38.907615 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:38.907521 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:38.929635 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:38.929591 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:39.104739 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.076055 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:39.136019 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.135977 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 15:30:39.162844 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.162750 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:39.168183 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.168146 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:30:39.191830 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.191782 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" podUID="12553958-5457-4c66-b011-431e954fef89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:39.277452 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.277414 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk"] Nov 03 15:30:39.301319 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.283522 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:39.371730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.346819 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:30:39.372017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.371976 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:39.407725 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.382967 4290 cpu_manager.go:414] "containerMap: removing container" podUID="d955c5f2-5757-4b5e-9667-9e3db67795bf" containerName="revoke-keys" Nov 03 15:30:39.501531 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.501438 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d955c5f2-5757-4b5e-9667-9e3db67795bf" containerName="revoke-keys" Nov 03 15:30:39.502106 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.502080 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="d955c5f2-5757-4b5e-9667-9e3db67795bf" containerName="revoke-keys" Nov 03 15:30:39.613264 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.596361 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:39.665788 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:39.659222 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:30:39.798826 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.798793 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk"] Nov 03 15:30:39.818948 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.818905 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" Nov 03 15:30:39.850855 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.850818 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:39.991227 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:39.991159 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:40.019364 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.019331 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-d785c53b\"" Nov 03 15:30:40.664770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.019827 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:40.664770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.020216 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:40.664770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.090746 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlwjn\" (UniqueName: \"kubernetes.io/projected/c1bbe6d2-f247-49f2-83ff-097721fcacbd-kube-api-access-qlwjn\") pod \"payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk\" (UID: \"c1bbe6d2-f247-49f2-83ff-097721fcacbd\") " pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" Nov 03 15:30:40.664770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.664047 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1bbe6d2-f247-49f2-83ff-097721fcacbd-plugins-config-volume\") pod \"payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk\" (UID: \"c1bbe6d2-f247-49f2-83ff-097721fcacbd\") " pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" Nov 03 15:30:40.664770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.182919 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:40.751896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.751836 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf"] Nov 03 15:30:40.783598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.262061 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 15:30:40.811755 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.808825 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 15:30:40.811755 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.374850 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:40.811755 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.538664 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:30:40.811755 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.809980 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:40.859467 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.859409 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Nov 03 15:30:40.859964 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.335982 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 15:30:40.882521 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.114480 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-d785c53b-dockercfg-cdg4j\"" Nov 03 15:30:40.882887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.216821 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded" Nov 03 15:30:40.979729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.885112 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlwjn\" (UniqueName: \"kubernetes.io/projected/c1bbe6d2-f247-49f2-83ff-097721fcacbd-kube-api-access-qlwjn\") pod \"payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk\" (UID: \"c1bbe6d2-f247-49f2-83ff-097721fcacbd\") " pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" Nov 03 15:30:40.980550 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:40.980501 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1bbe6d2-f247-49f2-83ff-097721fcacbd-plugins-config-volume\") pod \"payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk\" (UID: \"c1bbe6d2-f247-49f2-83ff-097721fcacbd\") " pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" Nov 03 15:30:41.030875 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.022672 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:41.117575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.042238 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1bbe6d2-f247-49f2-83ff-097721fcacbd-plugins-config-volume\") pod \"payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk\" (UID: \"c1bbe6d2-f247-49f2-83ff-097721fcacbd\") " pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" Nov 03 15:30:41.156835 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:41.150917 4290 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24535df2_4641_492e_aa5f_9bcdf1273d64.slice/crio-1c72926dda89df4304626630fd477598f9183f2d096bc266174cab3dc676dc71.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b80c4ee_4cf9_4cde_9e2f_1d4d72f0319e.slice/crio-conmon-329b780fceb9d3e4217873ee3192ff75c2a45244adc17de72892b323f0f759ac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b80c4ee_4cf9_4cde_9e2f_1d4d72f0319e.slice/crio-329b780fceb9d3e4217873ee3192ff75c2a45244adc17de72892b323f0f759ac.scope\": RecentStats: unable to find data in memory cache]" Nov 03 15:30:41.219461 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:30:41.201019 4290 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24535df2_4641_492e_aa5f_9bcdf1273d64.slice/crio-1c72926dda89df4304626630fd477598f9183f2d096bc266174cab3dc676dc71.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b80c4ee_4cf9_4cde_9e2f_1d4d72f0319e.slice/crio-conmon-329b780fceb9d3e4217873ee3192ff75c2a45244adc17de72892b323f0f759ac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24535df2_4641_492e_aa5f_9bcdf1273d64.slice/crio-conmon-1c72926dda89df4304626630fd477598f9183f2d096bc266174cab3dc676dc71.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b80c4ee_4cf9_4cde_9e2f_1d4d72f0319e.slice/crio-329b780fceb9d3e4217873ee3192ff75c2a45244adc17de72892b323f0f759ac.scope\": RecentStats: unable to find data in memory cache]" Nov 03 15:30:41.219461 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.215269 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlwjn\" (UniqueName: \"kubernetes.io/projected/c1bbe6d2-f247-49f2-83ff-097721fcacbd-kube-api-access-qlwjn\") pod \"payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk\" (UID: \"c1bbe6d2-f247-49f2-83ff-097721fcacbd\") " pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" Nov 03 15:30:41.261294 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.261255 4290 generic.go:356] "Generic (PLEG): container finished" podID="24535df2-4641-492e-aa5f-9bcdf1273d64" containerID="1c72926dda89df4304626630fd477598f9183f2d096bc266174cab3dc676dc71" exitCode=0 Nov 03 15:30:41.281331 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.281295 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" event={"ID":"24535df2-4641-492e-aa5f-9bcdf1273d64","Type":"ContainerDied","Data":"1c72926dda89df4304626630fd477598f9183f2d096bc266174cab3dc676dc71"} Nov 03 15:30:41.281606 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.281585 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf"] Nov 03 15:30:41.295623 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.289222 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" Nov 03 15:30:41.367170 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.349831 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" Nov 03 15:30:41.367170 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.364158 4290 generic.go:356] "Generic (PLEG): container finished" podID="3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" containerID="329b780fceb9d3e4217873ee3192ff75c2a45244adc17de72892b323f0f759ac" exitCode=0 Nov 03 15:30:41.413419 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.413342 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" event={"ID":"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e","Type":"ContainerDied","Data":"329b780fceb9d3e4217873ee3192ff75c2a45244adc17de72892b323f0f759ac"} Nov 03 15:30:41.413739 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.413717 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:41.415343 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.415327 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" Nov 03 15:30:41.419467 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.419416 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:30:41.426979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.426897 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz4qp\" (UniqueName: \"kubernetes.io/projected/33764de5-4ee3-4c10-b63c-17b701bca3bd-kube-api-access-hz4qp\") pod \"payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf\" (UID: \"33764de5-4ee3-4c10-b63c-17b701bca3bd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" Nov 03 15:30:41.428395 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.427187 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33764de5-4ee3-4c10-b63c-17b701bca3bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf\" (UID: \"33764de5-4ee3-4c10-b63c-17b701bca3bd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" Nov 03 15:30:41.431092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.431067 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:30:41.435518 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.435494 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:30:41.436825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.435790 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:30:41.487111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.445257 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:30:41.487111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.445362 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:30:41.487111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.445404 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:30:41.487111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.445439 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:30:41.487111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.451584 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:30:41.499227 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.499194 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:30:41.499403 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.499289 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" podStartSLOduration=18.499273247 podStartE2EDuration="18.499273247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:37.211897709 +0000 UTC m=+1344.642855516" watchObservedRunningTime="2026-11-03 15:30:41.499273247 +0000 UTC m=+1348.930231055" Nov 03 15:30:41.529184 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.529148 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-workload-certs\") pod \"24535df2-4641-492e-aa5f-9bcdf1273d64\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " Nov 03 15:30:41.529329 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.529212 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-token\" (UniqueName: \"kubernetes.io/projected/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-token\") pod \"24535df2-4641-492e-aa5f-9bcdf1273d64\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " Nov 03 15:30:41.529329 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.529284 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-workload-socket\") pod \"24535df2-4641-492e-aa5f-9bcdf1273d64\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " Nov 03 15:30:41.529431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.529334 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-envoy\") pod \"24535df2-4641-492e-aa5f-9bcdf1273d64\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " Nov 03 15:30:41.529431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.529372 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24535df2-4641-492e-aa5f-9bcdf1273d64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24535df2-4641-492e-aa5f-9bcdf1273d64-istiod-ca-cert\") pod \"24535df2-4641-492e-aa5f-9bcdf1273d64\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " Nov 03 15:30:41.529596 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.529575 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-data\") pod \"24535df2-4641-492e-aa5f-9bcdf1273d64\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " Nov 03 15:30:41.529652 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.529613 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24535df2-4641-492e-aa5f-9bcdf1273d64-kube-api-access-nrpxg\" (UniqueName: \"kubernetes.io/projected/24535df2-4641-492e-aa5f-9bcdf1273d64-kube-api-access-nrpxg\") pod \"24535df2-4641-492e-aa5f-9bcdf1273d64\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " Nov 03 15:30:41.529802 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.529782 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-podinfo\") pod \"24535df2-4641-492e-aa5f-9bcdf1273d64\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " Nov 03 15:30:41.529870 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.529842 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-credential-socket\") pod \"24535df2-4641-492e-aa5f-9bcdf1273d64\" (UID: \"24535df2-4641-492e-aa5f-9bcdf1273d64\") " Nov 03 15:30:41.532933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.532218 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz4qp\" (UniqueName: \"kubernetes.io/projected/33764de5-4ee3-4c10-b63c-17b701bca3bd-kube-api-access-hz4qp\") pod \"payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf\" (UID: \"33764de5-4ee3-4c10-b63c-17b701bca3bd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" Nov 03 15:30:41.532933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.532516 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33764de5-4ee3-4c10-b63c-17b701bca3bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf\" (UID: \"33764de5-4ee3-4c10-b63c-17b701bca3bd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" Nov 03 15:30:41.534938 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.534860 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24535df2-4641-492e-aa5f-9bcdf1273d64-istiod-ca-cert" pod "24535df2-4641-492e-aa5f-9bcdf1273d64" (UID: "24535df2-4641-492e-aa5f-9bcdf1273d64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:41.534938 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.534872 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24535df2-4641-492e-aa5f-9bcdf1273d64-kube-api-access-nrpxg" pod "24535df2-4641-492e-aa5f-9bcdf1273d64" (UID: "24535df2-4641-492e-aa5f-9bcdf1273d64"). InnerVolumeSpecName "kube-api-access-nrpxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:41.535110 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.535067 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-data" pod "24535df2-4641-492e-aa5f-9bcdf1273d64" (UID: "24535df2-4641-492e-aa5f-9bcdf1273d64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:30:41.535169 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.535125 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-workload-socket" pod "24535df2-4641-492e-aa5f-9bcdf1273d64" (UID: "24535df2-4641-492e-aa5f-9bcdf1273d64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:30:41.535724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.535307 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-workload-certs" pod "24535df2-4641-492e-aa5f-9bcdf1273d64" (UID: "24535df2-4641-492e-aa5f-9bcdf1273d64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:30:41.535724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.535599 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-credential-socket" pod "24535df2-4641-492e-aa5f-9bcdf1273d64" (UID: "24535df2-4641-492e-aa5f-9bcdf1273d64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:30:41.536286 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.535956 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33764de5-4ee3-4c10-b63c-17b701bca3bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf\" (UID: \"33764de5-4ee3-4c10-b63c-17b701bca3bd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" Nov 03 15:30:41.536286 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.536247 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-podinfo" pod "24535df2-4641-492e-aa5f-9bcdf1273d64" (UID: "24535df2-4641-492e-aa5f-9bcdf1273d64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:30:41.539478 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.539451 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-envoy" pod "24535df2-4641-492e-aa5f-9bcdf1273d64" (UID: "24535df2-4641-492e-aa5f-9bcdf1273d64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:30:41.540985 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.540951 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-token" pod "24535df2-4641-492e-aa5f-9bcdf1273d64" (UID: "24535df2-4641-492e-aa5f-9bcdf1273d64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:41.581598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.581547 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" podStartSLOduration=18.581532707 podStartE2EDuration="18.581532707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:39.455021297 +0000 UTC m=+1346.885979103" watchObservedRunningTime="2026-11-03 15:30:41.581532707 +0000 UTC m=+1349.012490512" Nov 03 15:30:41.584728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.584686 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:30:41.598910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.598879 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz4qp\" (UniqueName: \"kubernetes.io/projected/33764de5-4ee3-4c10-b63c-17b701bca3bd-kube-api-access-hz4qp\") pod \"payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf\" (UID: \"33764de5-4ee3-4c10-b63c-17b701bca3bd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" Nov 03 15:30:41.633809 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.633681 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-workload-socket\") pod \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " Nov 03 15:30:41.633809 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.633780 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-podinfo\") pod \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " Nov 03 15:30:41.634050 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.633829 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-token\" (UniqueName: \"kubernetes.io/projected/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-token\") pod \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " Nov 03 15:30:41.634050 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.633859 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-envoy\") pod \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " Nov 03 15:30:41.634050 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.633924 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-workload-certs\") pod \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " Nov 03 15:30:41.634050 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.633964 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-kube-api-access-vp8wp\" (UniqueName: \"kubernetes.io/projected/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-kube-api-access-vp8wp\") pod \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " Nov 03 15:30:41.634050 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.633963 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-workload-socket" pod "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" (UID: "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:30:41.634050 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.634013 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-data\") pod \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " Nov 03 15:30:41.634320 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.634052 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istiod-ca-cert\") pod \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " Nov 03 15:30:41.635126 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.634882 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istiod-ca-cert" pod "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" (UID: "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:41.635465 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.635289 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-data" pod "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" (UID: "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:30:41.635912 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.635600 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-workload-certs" pod "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" (UID: "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:30:41.635912 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.635650 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-credential-socket\") pod \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\" (UID: \"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e\") " Nov 03 15:30:41.636982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.636780 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.636982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.636806 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrpxg\" (UniqueName: \"kubernetes.io/projected/24535df2-4641-492e-aa5f-9bcdf1273d64-kube-api-access-nrpxg\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.636982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.636822 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.636982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.636818 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-envoy" pod "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" (UID: "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:30:41.636982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.636837 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.636982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.636851 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.636982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.636865 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.636982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.636879 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.636982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.636894 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.636982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.636907 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.636982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.636919 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.636982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.636933 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.636982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.636946 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24535df2-4641-492e-aa5f-9bcdf1273d64-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.636982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.636961 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24535df2-4641-492e-aa5f-9bcdf1273d64-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.636982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.636973 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-credential-socket" pod "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" (UID: "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:30:41.639919 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.639893 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-token" pod "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" (UID: "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:41.640776 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.640741 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-podinfo" pod "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" (UID: "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:30:41.641094 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.641068 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-kube-api-access-vp8wp" pod "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" (UID: "3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e"). InnerVolumeSpecName "kube-api-access-vp8wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:41.708762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.708650 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" Nov 03 15:30:41.739194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.739146 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.739194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.739185 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.739194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.739201 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.739401 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.739216 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.739401 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.739231 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp8wp\" (UniqueName: \"kubernetes.io/projected/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e-kube-api-access-vp8wp\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:41.833576 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:30:41.833549 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bbe6d2_f247_49f2_83ff_097721fcacbd.slice/crio-53024a32276a7264eb05aa944abfa5c9315367dd472c7a41f1a6d474ff5e6a92 WatchSource:0}: Error finding container 53024a32276a7264eb05aa944abfa5c9315367dd472c7a41f1a6d474ff5e6a92: Status 404 returned error can't find the container with id 53024a32276a7264eb05aa944abfa5c9315367dd472c7a41f1a6d474ff5e6a92 Nov 03 15:30:41.834623 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:41.834596 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk"] Nov 03 15:30:42.120959 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:30:42.120917 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33764de5_4ee3_4c10_b63c_17b701bca3bd.slice/crio-c5e83c22e8d83908e03f54a6113d46d3a61d234dcf119c2c5e8fe9df6fa24ba7 WatchSource:0}: Error finding container c5e83c22e8d83908e03f54a6113d46d3a61d234dcf119c2c5e8fe9df6fa24ba7: Status 404 returned error can't find the container with id c5e83c22e8d83908e03f54a6113d46d3a61d234dcf119c2c5e8fe9df6fa24ba7 Nov 03 15:30:42.123136 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:42.123102 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf"] Nov 03 15:30:42.402177 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:42.402141 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" event={"ID":"24535df2-4641-492e-aa5f-9bcdf1273d64","Type":"ContainerDied","Data":"9df219d6edc4a8a788339df0974d34089b6a5864755411f38fd919ed43947950"} Nov 03 15:30:42.402346 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:42.402228 4290 scope.go:122] "RemoveContainer" containerID="1c72926dda89df4304626630fd477598f9183f2d096bc266174cab3dc676dc71" Nov 03 15:30:42.402403 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:42.402368 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5" Nov 03 15:30:42.406846 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:42.406431 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" event={"ID":"3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e","Type":"ContainerDied","Data":"a6219826464706f530fe67c18681fb36cad4828ab108430a70151e494654d542"} Nov 03 15:30:42.406846 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:42.406830 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz" Nov 03 15:30:42.409201 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:42.409171 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" event={"ID":"33764de5-4ee3-4c10-b63c-17b701bca3bd","Type":"ContainerStarted","Data":"f0824e47100351847627ec42388ee95044be50e6bc997da67f200fecd153fed8"} Nov 03 15:30:42.409326 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:42.409220 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" event={"ID":"33764de5-4ee3-4c10-b63c-17b701bca3bd","Type":"ContainerStarted","Data":"c5e83c22e8d83908e03f54a6113d46d3a61d234dcf119c2c5e8fe9df6fa24ba7"} Nov 03 15:30:42.409912 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:42.409889 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" Nov 03 15:30:42.411979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:42.411956 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" event={"ID":"c1bbe6d2-f247-49f2-83ff-097721fcacbd","Type":"ContainerStarted","Data":"e06ac7f9a91c7c671fb9c2f0791f29615c7c55d25bc1dc90fe58467e5920f63a"} Nov 03 15:30:42.412172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:42.412142 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" event={"ID":"c1bbe6d2-f247-49f2-83ff-097721fcacbd","Type":"ContainerStarted","Data":"53024a32276a7264eb05aa944abfa5c9315367dd472c7a41f1a6d474ff5e6a92"} Nov 03 15:30:42.412303 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:42.412289 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" Nov 03 15:30:42.429511 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:42.429488 4290 scope.go:122] "RemoveContainer" containerID="329b780fceb9d3e4217873ee3192ff75c2a45244adc17de72892b323f0f759ac" Nov 03 15:30:42.913940 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:42.913880 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" podStartSLOduration=9.913867067 podStartE2EDuration="9.913867067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:42.912073871 +0000 UTC m=+1350.343031701" watchObservedRunningTime="2026-11-03 15:30:42.913867067 +0000 UTC m=+1350.344824872" Nov 03 15:30:43.174079 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:43.173960 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" podStartSLOduration=10.173943504 podStartE2EDuration="10.173943504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:43.046447521 +0000 UTC m=+1350.477405326" watchObservedRunningTime="2026-11-03 15:30:43.173943504 +0000 UTC m=+1350.604901311" Nov 03 15:30:43.236260 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:43.236224 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz"] Nov 03 15:30:43.243519 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:43.243490 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nm4sz"] Nov 03 15:30:43.446491 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:43.446402 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5"] Nov 03 15:30:43.478363 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:43.478330 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lcdb5"] Nov 03 15:30:44.693336 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:44.693301 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24535df2-4641-492e-aa5f-9bcdf1273d64" path="/var/lib/kubelet/pods/24535df2-4641-492e-aa5f-9bcdf1273d64/volumes" Nov 03 15:30:44.694099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:44.694078 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" path="/var/lib/kubelet/pods/3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e/volumes" Nov 03 15:30:45.907900 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:45.907861 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7"] Nov 03 15:30:45.908796 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:45.908776 4290 cpu_manager.go:414] "containerMap: removing container" podUID="24535df2-4641-492e-aa5f-9bcdf1273d64" containerName="istio-proxy" Nov 03 15:30:45.908863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:45.908796 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24535df2-4641-492e-aa5f-9bcdf1273d64" containerName="istio-proxy" Nov 03 15:30:45.908863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:45.908817 4290 cpu_manager.go:414] "containerMap: removing container" podUID="3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" containerName="istio-proxy" Nov 03 15:30:45.908863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:45.908822 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" containerName="istio-proxy" Nov 03 15:30:45.908957 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:45.908929 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="24535df2-4641-492e-aa5f-9bcdf1273d64" containerName="istio-proxy" Nov 03 15:30:45.908957 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:45.908942 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b80c4ee-4cf9-4cde-9e2f-1d4d72f0319e" containerName="istio-proxy" Nov 03 15:30:45.912605 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:45.912586 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" Nov 03 15:30:45.965594 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:45.965556 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cceaf27f\"" Nov 03 15:30:45.965841 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:45.965721 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-cceaf27f-dockercfg-825kd\"" Nov 03 15:30:45.968619 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:45.968590 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7"] Nov 03 15:30:46.033753 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.033712 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a549ce-c7d4-4c27-96a1-2a030683b871-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7\" (UID: \"a9a549ce-c7d4-4c27-96a1-2a030683b871\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" Nov 03 15:30:46.034341 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.033837 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2kck\" (UniqueName: \"kubernetes.io/projected/a9a549ce-c7d4-4c27-96a1-2a030683b871-kube-api-access-k2kck\") pod \"payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7\" (UID: \"a9a549ce-c7d4-4c27-96a1-2a030683b871\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" Nov 03 15:30:46.048862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.048828 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl"] Nov 03 15:30:46.053271 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.053246 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" Nov 03 15:30:46.125619 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.125582 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl"] Nov 03 15:30:46.144961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.144916 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65hm4\" (UniqueName: \"kubernetes.io/projected/2417d881-270c-4ce5-a27f-fa8737bd56eb-kube-api-access-65hm4\") pod \"payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl\" (UID: \"2417d881-270c-4ce5-a27f-fa8737bd56eb\") " pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" Nov 03 15:30:46.145361 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.145338 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2417d881-270c-4ce5-a27f-fa8737bd56eb-plugins-config-volume\") pod \"payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl\" (UID: \"2417d881-270c-4ce5-a27f-fa8737bd56eb\") " pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" Nov 03 15:30:46.145560 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.145545 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a549ce-c7d4-4c27-96a1-2a030683b871-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7\" (UID: \"a9a549ce-c7d4-4c27-96a1-2a030683b871\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" Nov 03 15:30:46.146632 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.146502 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a549ce-c7d4-4c27-96a1-2a030683b871-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7\" (UID: \"a9a549ce-c7d4-4c27-96a1-2a030683b871\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" Nov 03 15:30:46.146885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.146866 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2kck\" (UniqueName: \"kubernetes.io/projected/a9a549ce-c7d4-4c27-96a1-2a030683b871-kube-api-access-k2kck\") pod \"payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7\" (UID: \"a9a549ce-c7d4-4c27-96a1-2a030683b871\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" Nov 03 15:30:46.214310 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.214237 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2kck\" (UniqueName: \"kubernetes.io/projected/a9a549ce-c7d4-4c27-96a1-2a030683b871-kube-api-access-k2kck\") pod \"payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7\" (UID: \"a9a549ce-c7d4-4c27-96a1-2a030683b871\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" Nov 03 15:30:46.227477 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.227447 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" Nov 03 15:30:46.248926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.248899 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65hm4\" (UniqueName: \"kubernetes.io/projected/2417d881-270c-4ce5-a27f-fa8737bd56eb-kube-api-access-65hm4\") pod \"payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl\" (UID: \"2417d881-270c-4ce5-a27f-fa8737bd56eb\") " pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" Nov 03 15:30:46.249048 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.248953 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2417d881-270c-4ce5-a27f-fa8737bd56eb-plugins-config-volume\") pod \"payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl\" (UID: \"2417d881-270c-4ce5-a27f-fa8737bd56eb\") " pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" Nov 03 15:30:46.250216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.250193 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2417d881-270c-4ce5-a27f-fa8737bd56eb-plugins-config-volume\") pod \"payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl\" (UID: \"2417d881-270c-4ce5-a27f-fa8737bd56eb\") " pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" Nov 03 15:30:46.316760 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.313924 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65hm4\" (UniqueName: \"kubernetes.io/projected/2417d881-270c-4ce5-a27f-fa8737bd56eb-kube-api-access-65hm4\") pod \"payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl\" (UID: \"2417d881-270c-4ce5-a27f-fa8737bd56eb\") " pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" Nov 03 15:30:46.382249 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.381308 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx"] Nov 03 15:30:46.382249 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.381458 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" Nov 03 15:30:46.390757 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.387956 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.461749 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.460163 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzc7q\" (UniqueName: \"kubernetes.io/projected/9d036ddd-5485-4456-986e-a10a2d8c286b-kube-api-access-nzc7q\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.461749 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.460285 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d036ddd-5485-4456-986e-a10a2d8c286b-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.461749 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.460375 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.461749 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.460494 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.461749 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.460551 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.461749 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.460582 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.461749 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.460617 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.461749 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.460659 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.461749 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.460770 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.495766 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.493574 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx"] Nov 03 15:30:46.564208 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.562682 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d036ddd-5485-4456-986e-a10a2d8c286b-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.564547 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.564519 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.564686 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.564651 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.564799 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.564753 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.564799 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.564784 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.564918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.564817 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.564918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.564859 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.565022 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.564950 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.565022 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.564969 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d036ddd-5485-4456-986e-a10a2d8c286b-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.565133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.565070 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzc7q\" (UniqueName: \"kubernetes.io/projected/9d036ddd-5485-4456-986e-a10a2d8c286b-kube-api-access-nzc7q\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.565933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.565286 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.565933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.565622 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.565933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.565889 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.566862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.566372 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.569870 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.569840 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.570687 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.570660 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.629139 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.628687 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.629490 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.629463 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzc7q\" (UniqueName: \"kubernetes.io/projected/9d036ddd-5485-4456-986e-a10a2d8c286b-kube-api-access-nzc7q\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.631981 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:30:46.631950 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a549ce_c7d4_4c27_96a1_2a030683b871.slice/crio-fa49503b212bf84c55c6c3af44406ddec3ba14f4f284dc7cd9d361012f1cd870 WatchSource:0}: Error finding container fa49503b212bf84c55c6c3af44406ddec3ba14f4f284dc7cd9d361012f1cd870: Status 404 returned error can't find the container with id fa49503b212bf84c55c6c3af44406ddec3ba14f4f284dc7cd9d361012f1cd870 Nov 03 15:30:46.633364 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.633341 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7"] Nov 03 15:30:46.731560 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.731531 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:46.818651 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.818406 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl"] Nov 03 15:30:46.879883 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.879831 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh"] Nov 03 15:30:46.885482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.885460 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:46.945046 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.945017 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh"] Nov 03 15:30:46.972062 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.972021 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:46.972249 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.972129 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:46.972249 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.972211 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:46.972249 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.972242 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:46.972404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.972306 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:46.972404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.972353 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:46.972404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.972382 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:46.972404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.972398 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:46.972601 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:46.972453 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmbtb\" (UniqueName: \"kubernetes.io/projected/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-kube-api-access-xmbtb\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.074078 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.074005 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.074078 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.074039 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.074269 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.074107 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmbtb\" (UniqueName: \"kubernetes.io/projected/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-kube-api-access-xmbtb\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.074322 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.074276 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.074394 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.074363 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.074454 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.074438 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.074501 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.074465 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.074548 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.074511 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.074548 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.074513 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.074645 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.074609 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.074645 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.074622 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.074767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.074737 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.074856 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.074832 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.075287 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.075263 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.077070 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.077047 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.077142 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.077098 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.137190 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:30:47.137159 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d036ddd_5485_4456_986e_a10a2d8c286b.slice/crio-e86a829bdc7e50a4a9e0ba60a4808f407cd0ed14f396a8a39de3d412e794ecc4 WatchSource:0}: Error finding container e86a829bdc7e50a4a9e0ba60a4808f407cd0ed14f396a8a39de3d412e794ecc4: Status 404 returned error can't find the container with id e86a829bdc7e50a4a9e0ba60a4808f407cd0ed14f396a8a39de3d412e794ecc4 Nov 03 15:30:47.138602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.138571 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx"] Nov 03 15:30:47.139626 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.139607 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.139626 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.139617 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmbtb\" (UniqueName: \"kubernetes.io/projected/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-kube-api-access-xmbtb\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.140483 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.140448 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:47.140591 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.140513 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:47.140591 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.140544 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:47.204946 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.204921 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:47.497403 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.497369 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" event={"ID":"9d036ddd-5485-4456-986e-a10a2d8c286b","Type":"ContainerStarted","Data":"733ebefe5736b008c3f8de9ed217c9ed697d7458a2db0fa9d9378c31bcf6b010"} Nov 03 15:30:47.497587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.497424 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" event={"ID":"9d036ddd-5485-4456-986e-a10a2d8c286b","Type":"ContainerStarted","Data":"e86a829bdc7e50a4a9e0ba60a4808f407cd0ed14f396a8a39de3d412e794ecc4"} Nov 03 15:30:47.499317 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.499288 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" event={"ID":"2417d881-270c-4ce5-a27f-fa8737bd56eb","Type":"ContainerStarted","Data":"7a67e4eb525d82f3cac5f890ff241d5075e86d8c5bb3ff5244cec54c7b8c7044"} Nov 03 15:30:47.499443 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.499343 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" event={"ID":"2417d881-270c-4ce5-a27f-fa8737bd56eb","Type":"ContainerStarted","Data":"3cbee401729b2961d3c68afa8b70df2f59b91812b54f15efcc9301588841961d"} Nov 03 15:30:47.499443 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.499376 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" Nov 03 15:30:47.501133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.501103 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" event={"ID":"a9a549ce-c7d4-4c27-96a1-2a030683b871","Type":"ContainerStarted","Data":"26ce160af85ab86a97429a4d2e9ba641c2149b9f3aa4af2e395fa951a0d19551"} Nov 03 15:30:47.501251 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.501154 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" event={"ID":"a9a549ce-c7d4-4c27-96a1-2a030683b871","Type":"ContainerStarted","Data":"fa49503b212bf84c55c6c3af44406ddec3ba14f4f284dc7cd9d361012f1cd870"} Nov 03 15:30:47.501251 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.501211 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" Nov 03 15:30:47.619396 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.619335 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh"] Nov 03 15:30:47.619776 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.619720 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" podStartSLOduration=1.619681436 podStartE2EDuration="1.619681436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:47.617880838 +0000 UTC m=+1355.048838646" watchObservedRunningTime="2026-11-03 15:30:47.619681436 +0000 UTC m=+1355.050639243" Nov 03 15:30:47.620359 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:30:47.620305 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafad8a84_b2c2_4ab6_8e5c_9d6f67779ec8.slice/crio-36035fd08223b9e46fdeb0e6c4d38a7675e3050a2fe10f3d22ba047dc68650c5 WatchSource:0}: Error finding container 36035fd08223b9e46fdeb0e6c4d38a7675e3050a2fe10f3d22ba047dc68650c5: Status 404 returned error can't find the container with id 36035fd08223b9e46fdeb0e6c4d38a7675e3050a2fe10f3d22ba047dc68650c5 Nov 03 15:30:47.625325 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.625273 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:47.625553 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.625521 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:47.625647 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.625591 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:47.731988 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.731961 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:47.750781 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.749218 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" podStartSLOduration=1.749194761 podStartE2EDuration="1.749194761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:47.734045134 +0000 UTC m=+1355.165002944" watchObservedRunningTime="2026-11-03 15:30:47.749194761 +0000 UTC m=+1355.180152569" Nov 03 15:30:47.902419 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:47.900890 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" podStartSLOduration=2.9008703970000003 podStartE2EDuration="2.900870397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:47.860313801 +0000 UTC m=+1355.291271633" watchObservedRunningTime="2026-11-03 15:30:47.900870397 +0000 UTC m=+1355.331828205" Nov 03 15:30:48.520466 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:48.519948 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" event={"ID":"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8","Type":"ContainerStarted","Data":"321dfb447dbbfe54fa34c09ebb256c6374f30712ab8cc0c712b5cd25dc1334e8"} Nov 03 15:30:48.520466 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:48.520015 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" event={"ID":"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8","Type":"ContainerStarted","Data":"36035fd08223b9e46fdeb0e6c4d38a7675e3050a2fe10f3d22ba047dc68650c5"} Nov 03 15:30:48.710255 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:48.710190 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" podStartSLOduration=2.710172379 podStartE2EDuration="2.710172379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:48.692865034 +0000 UTC m=+1356.123822840" watchObservedRunningTime="2026-11-03 15:30:48.710172379 +0000 UTC m=+1356.141130186" Nov 03 15:30:48.733644 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:48.733615 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 15:30:48.733819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:48.733664 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" podUID="9d036ddd-5485-4456-986e-a10a2d8c286b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 15:30:49.205388 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:49.205353 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:49.207038 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:49.207012 4290 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 15:30:49.207170 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:49.207075 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" podUID="afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 15:30:49.733456 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:49.733423 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 15:30:49.733991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:49.733492 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" podUID="9d036ddd-5485-4456-986e-a10a2d8c286b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 15:30:50.205944 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:50.205912 4290 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 15:30:50.206165 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:50.206137 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" podUID="afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 15:30:50.732335 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:50.732282 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 15:30:50.732482 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:50.732347 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" podUID="9d036ddd-5485-4456-986e-a10a2d8c286b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 15:30:51.205420 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:51.205387 4290 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 15:30:51.205851 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:51.205435 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" podUID="afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 15:30:51.732746 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:51.732679 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 15:30:51.732997 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:51.732778 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" podUID="9d036ddd-5485-4456-986e-a10a2d8c286b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 15:30:52.205349 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:52.205316 4290 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 15:30:52.205560 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:52.205382 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" podUID="afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 15:30:52.732312 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:52.732266 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 15:30:52.732480 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:52.732334 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" podUID="9d036ddd-5485-4456-986e-a10a2d8c286b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 15:30:53.205475 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:53.205438 4290 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 15:30:53.205653 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:53.205510 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" podUID="afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 15:30:53.436713 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:53.436681 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" Nov 03 15:30:53.732650 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:53.732615 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 15:30:53.732871 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:53.732665 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" podUID="9d036ddd-5485-4456-986e-a10a2d8c286b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 15:30:54.206180 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.206142 4290 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 15:30:54.206723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.206219 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" podUID="afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 15:30:54.358343 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.358305 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn"] Nov 03 15:30:54.364712 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.364679 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.418101 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.418072 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-th69q\"" Nov 03 15:30:54.418288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.418125 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 15:30:54.418288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.418088 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 15:30:54.418288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.418079 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 15:30:54.446189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.445499 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.446189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.445562 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.446189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.445750 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e53c44e-f134-4562-990b-19a120e0bef0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.446189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.445842 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e53c44e-f134-4562-990b-19a120e0bef0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.446189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.445889 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckchk\" (UniqueName: \"kubernetes.io/projected/9e53c44e-f134-4562-990b-19a120e0bef0-kube-api-access-ckchk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.446189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.446025 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e53c44e-f134-4562-990b-19a120e0bef0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.446189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.446055 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.455875 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.451177 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" Nov 03 15:30:54.477893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.477813 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn"] Nov 03 15:30:54.548726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.547718 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e53c44e-f134-4562-990b-19a120e0bef0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.548726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.547860 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e53c44e-f134-4562-990b-19a120e0bef0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.548726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.547918 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckchk\" (UniqueName: \"kubernetes.io/projected/9e53c44e-f134-4562-990b-19a120e0bef0-kube-api-access-ckchk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.548726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.548114 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e53c44e-f134-4562-990b-19a120e0bef0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.548726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.548152 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.548726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.548228 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.548726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.548269 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.565814 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.550805 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e53c44e-f134-4562-990b-19a120e0bef0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.565814 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.554196 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e53c44e-f134-4562-990b-19a120e0bef0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.565814 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.556518 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.565814 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.558997 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.575053 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.575016 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.615314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.615203 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckchk\" (UniqueName: \"kubernetes.io/projected/9e53c44e-f134-4562-990b-19a120e0bef0-kube-api-access-ckchk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.634782 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.629763 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e53c44e-f134-4562-990b-19a120e0bef0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xk5cn\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:54.688728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:54.681998 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:55.114876 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:55.113483 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:55.114876 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:55.114679 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:55.117133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:55.117108 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:30:55.207057 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:55.207011 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn"] Nov 03 15:30:55.544192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:55.544150 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:55.545671 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:55.545647 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:55.547299 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:55.547263 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:30:55.652099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:55.652061 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" event={"ID":"9e53c44e-f134-4562-990b-19a120e0bef0","Type":"ContainerStarted","Data":"7bd49c08cb6cfda1621bf17987a5aa7b8304e7fe8582d848494fea99c45d8a1d"} Nov 03 15:30:58.524499 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:58.524468 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" Nov 03 15:30:58.525065 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:58.524547 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" Nov 03 15:30:59.395679 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:59.395644 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:59.395812 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:59.395735 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:30:59.708730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:59.708615 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" event={"ID":"9e53c44e-f134-4562-990b-19a120e0bef0","Type":"ContainerStarted","Data":"f1015a57ceb279fe0efbd3090b531b221697287ef5cbfe047bf286121a123035"} Nov 03 15:30:59.708730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:59.708676 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:30:59.834379 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:30:59.834324 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" podStartSLOduration=1.651275321 podStartE2EDuration="5.834309306s" totalImagesPullingTime="4.183033985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:59.829361471 +0000 UTC m=+1367.260319277" watchObservedRunningTime="2026-11-03 15:30:59.834309306 +0000 UTC m=+1367.265267113" Nov 03 15:31:00.724042 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:00.724007 4290 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-xk5cn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:31:00.724480 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:00.724103 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" podUID="9e53c44e-f134-4562-990b-19a120e0bef0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:31:01.511090 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.511058 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp"] Nov 03 15:31:01.516046 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.516025 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.694191 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.694159 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d888abb-c170-4809-a0d3-642d20b77628-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.694369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.694290 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzxph\" (UniqueName: \"kubernetes.io/projected/5d888abb-c170-4809-a0d3-642d20b77628-kube-api-access-kzxph\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.694449 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.694373 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.694529 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.694502 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.694615 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.694598 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d888abb-c170-4809-a0d3-642d20b77628-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.694663 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.694642 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d888abb-c170-4809-a0d3-642d20b77628-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.694720 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.694682 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.694768 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.694755 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.694830 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.694818 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.732273 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.732236 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp"] Nov 03 15:31:01.796335 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.796256 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzxph\" (UniqueName: \"kubernetes.io/projected/5d888abb-c170-4809-a0d3-642d20b77628-kube-api-access-kzxph\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.796335 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.796305 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.796561 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.796369 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.796561 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.796415 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d888abb-c170-4809-a0d3-642d20b77628-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.796561 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.796447 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d888abb-c170-4809-a0d3-642d20b77628-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.796561 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.796485 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.796812 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.796573 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.796812 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.796663 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.796812 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.796765 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d888abb-c170-4809-a0d3-642d20b77628-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.796991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.796839 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.796991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.796873 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.797105 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.797051 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.797331 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.797306 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.797500 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.797482 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d888abb-c170-4809-a0d3-642d20b77628-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.799143 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.799118 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d888abb-c170-4809-a0d3-642d20b77628-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.799242 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.799118 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.858134 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.858105 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d888abb-c170-4809-a0d3-642d20b77628-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:01.859110 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:01.859086 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzxph\" (UniqueName: \"kubernetes.io/projected/5d888abb-c170-4809-a0d3-642d20b77628-kube-api-access-kzxph\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:02.131192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:02.131167 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:02.592204 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:02.592173 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp"] Nov 03 15:31:02.595933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:02.595899 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:31:02.596014 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:02.595988 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:31:02.596056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:02.596021 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:31:02.745208 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:02.745171 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" event={"ID":"5d888abb-c170-4809-a0d3-642d20b77628","Type":"ContainerStarted","Data":"de177b1f7f97a7c6bbb3c271eb4ca9bd44ef4011d62b408e72a46dde532e284d"} Nov 03 15:31:02.745622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:02.745234 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" event={"ID":"5d888abb-c170-4809-a0d3-642d20b77628","Type":"ContainerStarted","Data":"59137be573767137a481cba73252bf694430dc09c12cd7f3ceb55eb855bb0ff5"} Nov 03 15:31:02.882852 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:02.882799 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" podStartSLOduration=1.8827802409999999 podStartE2EDuration="1.882780241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:02.880695326 +0000 UTC m=+1370.311653130" watchObservedRunningTime="2026-11-03 15:31:02.882780241 +0000 UTC m=+1370.313738052" Nov 03 15:31:03.132188 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:03.132157 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:03.133513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:03.133449 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 15:31:03.133513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:03.133497 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 15:31:03.722674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:03.722642 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:31:04.132385 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:04.132353 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 15:31:04.132801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:04.132404 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 15:31:05.132127 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:05.132096 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 15:31:05.132292 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:05.132147 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 15:31:06.132480 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:06.132450 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 15:31:06.132949 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:06.132501 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 15:31:07.131550 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:07.131513 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 15:31:07.131762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:07.131571 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 15:31:08.132117 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:08.132078 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 15:31:08.132525 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:08.132163 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 15:31:08.730958 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:31:08.730855 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:31:09.131621 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:09.131584 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 15:31:09.131819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:09.131649 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 15:31:10.132619 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:10.132579 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 15:31:10.133237 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:10.133198 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 15:31:12.150745 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:12.141191 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:31:12.151460 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:12.151423 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:12.502464 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:12.501819 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:12.797843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:12.791468 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:12.913738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:12.911881 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Nov 03 15:31:13.085804 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:13.015544 4290 scope.go:122] "RemoveContainer" containerID="2d2c71cee789e7298c772347f11c1a542cac6dd97b51c72a8c61e3a93dcb96fc" Nov 03 15:31:13.117755 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:13.047168 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:13.117755 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:13.068202 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:13.375215 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:13.375184 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:31:13.375803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:13.375766 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:13.471870 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:13.471825 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:13.506585 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:13.506542 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:13.554759 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:13.552341 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:31:13.919779 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:13.870259 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:13.921581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:13.921555 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:31:13.924944 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:13.924913 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:13.941916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:13.926143 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:31:14.050730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:14.049803 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 15:31:14.118727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:14.108909 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:14.118924 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:14.118884 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:31:14.370840 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:14.368779 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:14.435732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:14.433749 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:31:14.435732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:14.433833 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:14.487411 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:14.475535 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:14.523741 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:14.511862 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Nov 03 15:31:14.538399 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:14.537783 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:31:14.926973 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:14.926932 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Nov 03 15:31:15.082287 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:15.082188 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 15:31:15.123728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:15.094128 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.123728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:15.120825 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 15:31:15.397472 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:15.377107 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 15:31:15.478569 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:15.478520 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 15:31:15.513723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:15.513664 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:31:15.513997 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:15.513981 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:31:15.542763 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:15.538850 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Nov 03 15:31:15.558815 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:15.558753 4290 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:31:15.567921 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:15.567808 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.685734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:15.673454 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:15.685734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:15.674633 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:15.714309 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:15.710858 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:31:16.086369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:16.086279 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:31:16.123758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:16.120005 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 15:31:16.279786 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:16.268810 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:16.279786 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:16.274229 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:31:16.382734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:16.379732 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:31:16.414727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:16.385446 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:31:16.481765 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:16.478010 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:16.618259 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:16.618232 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:31:16.666063 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:16.640161 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:31:17.030586 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:17.030478 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:31:17.110129 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:17.110065 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:29.738713 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.738626 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4"] Nov 03 15:31:29.743518 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.743495 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.814614 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.814581 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3cec29f-05df-404d-b346-53dda11adf8c-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.814798 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.814730 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldbtl\" (UniqueName: \"kubernetes.io/projected/a3cec29f-05df-404d-b346-53dda11adf8c-kube-api-access-ldbtl\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.814852 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.814814 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3cec29f-05df-404d-b346-53dda11adf8c-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.814898 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.814851 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3cec29f-05df-404d-b346-53dda11adf8c-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.814997 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.814974 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3cec29f-05df-404d-b346-53dda11adf8c-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.815066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.815053 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cec29f-05df-404d-b346-53dda11adf8c-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.815104 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.815080 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cec29f-05df-404d-b346-53dda11adf8c-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.815146 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.815111 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3cec29f-05df-404d-b346-53dda11adf8c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.815184 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.815154 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3cec29f-05df-404d-b346-53dda11adf8c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.916893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.916861 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3cec29f-05df-404d-b346-53dda11adf8c-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.917058 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.916954 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldbtl\" (UniqueName: \"kubernetes.io/projected/a3cec29f-05df-404d-b346-53dda11adf8c-kube-api-access-ldbtl\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.917058 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.916991 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3cec29f-05df-404d-b346-53dda11adf8c-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.917058 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.917022 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3cec29f-05df-404d-b346-53dda11adf8c-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.917180 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.917122 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3cec29f-05df-404d-b346-53dda11adf8c-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.917230 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.917214 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cec29f-05df-404d-b346-53dda11adf8c-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.917288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.917247 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3cec29f-05df-404d-b346-53dda11adf8c-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.917288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.917252 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cec29f-05df-404d-b346-53dda11adf8c-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.917399 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.917322 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3cec29f-05df-404d-b346-53dda11adf8c-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.917399 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.917341 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3cec29f-05df-404d-b346-53dda11adf8c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.917399 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.917373 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3cec29f-05df-404d-b346-53dda11adf8c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.917580 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.917558 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cec29f-05df-404d-b346-53dda11adf8c-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.917641 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.917630 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cec29f-05df-404d-b346-53dda11adf8c-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.918114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.918066 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3cec29f-05df-404d-b346-53dda11adf8c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.919612 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.919584 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3cec29f-05df-404d-b346-53dda11adf8c-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.919825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.919808 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3cec29f-05df-404d-b346-53dda11adf8c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.959296 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.959267 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4"] Nov 03 15:31:29.978827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.978795 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldbtl\" (UniqueName: \"kubernetes.io/projected/a3cec29f-05df-404d-b346-53dda11adf8c-kube-api-access-ldbtl\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:29.979051 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:29.979025 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3cec29f-05df-404d-b346-53dda11adf8c-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-wl2z4\" (UID: \"a3cec29f-05df-404d-b346-53dda11adf8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:30.058178 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:30.058107 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:30.510410 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:31:30.510362 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3cec29f_05df_404d_b346_53dda11adf8c.slice/crio-ffa12092efbe592d7384dab591ab88aaf6f7ac356425a1fbc37a2f8b3e5ef818 WatchSource:0}: Error finding container ffa12092efbe592d7384dab591ab88aaf6f7ac356425a1fbc37a2f8b3e5ef818: Status 404 returned error can't find the container with id ffa12092efbe592d7384dab591ab88aaf6f7ac356425a1fbc37a2f8b3e5ef818 Nov 03 15:31:30.512102 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:30.512023 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4"] Nov 03 15:31:30.513440 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:30.513386 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:31:30.513552 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:30.513482 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:31:30.513552 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:30.513520 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:31:31.276355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:31.276321 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" event={"ID":"a3cec29f-05df-404d-b346-53dda11adf8c","Type":"ContainerStarted","Data":"4e1631087692757879a8b8793929f2663df1fc6b1f47529439176c4bd26b141c"} Nov 03 15:31:31.276861 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:31.276372 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" event={"ID":"a3cec29f-05df-404d-b346-53dda11adf8c","Type":"ContainerStarted","Data":"ffa12092efbe592d7384dab591ab88aaf6f7ac356425a1fbc37a2f8b3e5ef818"} Nov 03 15:31:31.467072 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:31.466990 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" podStartSLOduration=2.46697101 podStartE2EDuration="2.46697101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:31.46443487 +0000 UTC m=+1398.895392687" watchObservedRunningTime="2026-11-03 15:31:31.46697101 +0000 UTC m=+1398.897928818" Nov 03 15:31:31.901354 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:31.901318 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk"] Nov 03 15:31:31.906969 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:31.906945 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:31.993724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:31.985936 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk"] Nov 03 15:31:32.061723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.061185 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:32.068773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.067809 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-wl2z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 15:31:32.068773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.067872 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" podUID="a3cec29f-05df-404d-b346-53dda11adf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 15:31:32.088185 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.088140 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.088369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.088215 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.088369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.088362 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.088533 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.088508 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2rj9\" (UniqueName: \"kubernetes.io/projected/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-kube-api-access-h2rj9\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.088600 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.088576 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.088659 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.088620 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.088659 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.088651 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.088802 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.088736 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.088859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.088788 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.190732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.190643 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.191155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.191127 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2rj9\" (UniqueName: \"kubernetes.io/projected/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-kube-api-access-h2rj9\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.191311 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.191295 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.191425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.191415 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.191508 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.191498 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.191585 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.191576 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.191644 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.191636 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.191789 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.191779 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.191891 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.191880 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.192203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.192133 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.192203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.192135 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.192366 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.192246 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.192728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.192688 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.193552 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.193529 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.195190 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.195165 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.196359 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.196335 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.273152 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.272964 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2rj9\" (UniqueName: \"kubernetes.io/projected/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-kube-api-access-h2rj9\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.273907 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.273883 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:32.524281 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:32.524201 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:33.021463 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:31:33.021432 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fda87ab_de4d_4bac_908f_b8c2ad6123e8.slice/crio-9c5ba570c891f675dc64ed8a36782c8e389dd394bc00bdcb9ee9cd3916993f5d WatchSource:0}: Error finding container 9c5ba570c891f675dc64ed8a36782c8e389dd394bc00bdcb9ee9cd3916993f5d: Status 404 returned error can't find the container with id 9c5ba570c891f675dc64ed8a36782c8e389dd394bc00bdcb9ee9cd3916993f5d Nov 03 15:31:33.023812 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:33.023783 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk"] Nov 03 15:31:33.024310 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:33.024275 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:31:33.024365 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:33.024342 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:31:33.024401 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:33.024371 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:31:33.060502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:33.060201 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-wl2z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 15:31:33.060502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:33.060265 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" podUID="a3cec29f-05df-404d-b346-53dda11adf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 15:31:33.306868 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:33.306779 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" event={"ID":"1fda87ab-de4d-4bac-908f-b8c2ad6123e8","Type":"ContainerStarted","Data":"c2eae0c017aa35ccd1476ae7115cfb7c4b86530e80951ebbdbec64b3c7c50974"} Nov 03 15:31:33.306868 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:33.306838 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" event={"ID":"1fda87ab-de4d-4bac-908f-b8c2ad6123e8","Type":"ContainerStarted","Data":"9c5ba570c891f675dc64ed8a36782c8e389dd394bc00bdcb9ee9cd3916993f5d"} Nov 03 15:31:33.437235 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:33.437185 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" podStartSLOduration=2.437170953 podStartE2EDuration="2.437170953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:33.435429246 +0000 UTC m=+1400.866387052" watchObservedRunningTime="2026-11-03 15:31:33.437170953 +0000 UTC m=+1400.868128758" Nov 03 15:31:33.525228 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:33.525193 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:34.068188 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:34.068148 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-wl2z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 15:31:34.068491 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:34.068457 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" podUID="a3cec29f-05df-404d-b346-53dda11adf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 15:31:34.526504 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:34.526292 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 15:31:34.526504 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:34.526356 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" podUID="1fda87ab-de4d-4bac-908f-b8c2ad6123e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 15:31:35.076726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:35.076156 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-wl2z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 15:31:35.076726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:35.076221 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" podUID="a3cec29f-05df-404d-b346-53dda11adf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 15:31:35.525736 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:35.525133 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 15:31:35.525736 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:35.525192 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" podUID="1fda87ab-de4d-4bac-908f-b8c2ad6123e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 15:31:36.062729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:36.061810 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-wl2z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 15:31:36.062729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:36.061874 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" podUID="a3cec29f-05df-404d-b346-53dda11adf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 15:31:36.529387 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:36.529352 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 15:31:36.529871 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:36.529832 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" podUID="1fda87ab-de4d-4bac-908f-b8c2ad6123e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 15:31:37.066108 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:37.066065 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-wl2z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 15:31:37.066616 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:37.066132 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" podUID="a3cec29f-05df-404d-b346-53dda11adf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 15:31:37.525372 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:37.525337 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 15:31:37.525591 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:37.525390 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" podUID="1fda87ab-de4d-4bac-908f-b8c2ad6123e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 15:31:38.065189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:38.065140 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-wl2z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 15:31:38.065379 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:38.065209 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" podUID="a3cec29f-05df-404d-b346-53dda11adf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 15:31:38.524539 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:38.524506 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 15:31:38.524975 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:38.524556 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" podUID="1fda87ab-de4d-4bac-908f-b8c2ad6123e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 15:31:38.712148 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:31:38.712104 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:31:39.534856 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:39.534822 4290 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 15:31:39.536321 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:39.534875 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" podUID="1fda87ab-de4d-4bac-908f-b8c2ad6123e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 15:31:39.735837 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:39.735807 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:39.737522 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:39.737499 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:39.738355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:39.738332 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" Nov 03 15:31:40.930655 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:40.930605 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:40.931300 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:40.931202 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:40.932159 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:40.932136 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:31:44.863130 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:44.863098 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp"] Nov 03 15:31:44.867231 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:44.867209 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:44.936936 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:44.936902 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56971b8-db79-464a-bdaa-1a1e950f7c80-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:44.937089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:44.936962 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56971b8-db79-464a-bdaa-1a1e950f7c80-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:44.937089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:44.937069 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56971b8-db79-464a-bdaa-1a1e950f7c80-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:44.937202 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:44.937185 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56971b8-db79-464a-bdaa-1a1e950f7c80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:44.937265 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:44.937254 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56971b8-db79-464a-bdaa-1a1e950f7c80-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:44.937301 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:44.937279 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56971b8-db79-464a-bdaa-1a1e950f7c80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:44.937446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:44.937432 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgcrp\" (UniqueName: \"kubernetes.io/projected/c56971b8-db79-464a-bdaa-1a1e950f7c80-kube-api-access-rgcrp\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:44.937481 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:44.937463 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56971b8-db79-464a-bdaa-1a1e950f7c80-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:44.937517 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:44.937489 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56971b8-db79-464a-bdaa-1a1e950f7c80-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.038648 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.038618 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56971b8-db79-464a-bdaa-1a1e950f7c80-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.038869 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.038668 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56971b8-db79-464a-bdaa-1a1e950f7c80-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.038869 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.038734 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56971b8-db79-464a-bdaa-1a1e950f7c80-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.038869 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.038803 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56971b8-db79-464a-bdaa-1a1e950f7c80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.038869 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.038861 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56971b8-db79-464a-bdaa-1a1e950f7c80-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.039114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.038879 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56971b8-db79-464a-bdaa-1a1e950f7c80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.039114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.039008 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgcrp\" (UniqueName: \"kubernetes.io/projected/c56971b8-db79-464a-bdaa-1a1e950f7c80-kube-api-access-rgcrp\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.039114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.039032 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56971b8-db79-464a-bdaa-1a1e950f7c80-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.039114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.039046 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56971b8-db79-464a-bdaa-1a1e950f7c80-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.039114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.039096 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56971b8-db79-464a-bdaa-1a1e950f7c80-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.039114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.039097 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56971b8-db79-464a-bdaa-1a1e950f7c80-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.039478 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.039435 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56971b8-db79-464a-bdaa-1a1e950f7c80-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.039608 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.039584 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56971b8-db79-464a-bdaa-1a1e950f7c80-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.039865 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.039845 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56971b8-db79-464a-bdaa-1a1e950f7c80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.041386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.041368 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56971b8-db79-464a-bdaa-1a1e950f7c80-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.041661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.041643 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56971b8-db79-464a-bdaa-1a1e950f7c80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.086580 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.086543 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp"] Nov 03 15:31:45.101438 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.101404 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56971b8-db79-464a-bdaa-1a1e950f7c80-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.102559 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.102534 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgcrp\" (UniqueName: \"kubernetes.io/projected/c56971b8-db79-464a-bdaa-1a1e950f7c80-kube-api-access-rgcrp\") pod \"maas-default-gateway-openshift-default-596dc9545f-pvhdp\" (UID: \"c56971b8-db79-464a-bdaa-1a1e950f7c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.183985 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.183955 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:45.634074 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:31:45.634044 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56971b8_db79_464a_bdaa_1a1e950f7c80.slice/crio-b0915ee93611f9ecfcc51d79ca30e3210bfd977ee7d910b0ef5335202297bbc1 WatchSource:0}: Error finding container b0915ee93611f9ecfcc51d79ca30e3210bfd977ee7d910b0ef5335202297bbc1: Status 404 returned error can't find the container with id b0915ee93611f9ecfcc51d79ca30e3210bfd977ee7d910b0ef5335202297bbc1 Nov 03 15:31:45.635598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.635576 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp"] Nov 03 15:31:45.636868 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.636838 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:31:45.636922 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.636900 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:31:45.636957 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:45.636928 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:31:46.614340 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:46.614305 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" event={"ID":"c56971b8-db79-464a-bdaa-1a1e950f7c80","Type":"ContainerStarted","Data":"77d5f96096e5ba2d5f4bffd57faaf55def81f094d0a8d8e24f02f506cd627479"} Nov 03 15:31:46.614862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:46.614372 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" event={"ID":"c56971b8-db79-464a-bdaa-1a1e950f7c80","Type":"ContainerStarted","Data":"b0915ee93611f9ecfcc51d79ca30e3210bfd977ee7d910b0ef5335202297bbc1"} Nov 03 15:31:46.874061 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:46.873991 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" podStartSLOduration=2.873969068 podStartE2EDuration="2.873969068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:46.786570085 +0000 UTC m=+1414.217527891" watchObservedRunningTime="2026-11-03 15:31:46.873969068 +0000 UTC m=+1414.304926874" Nov 03 15:31:47.184783 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:47.184669 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:47.186518 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:47.186488 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pvhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 15:31:47.186659 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:47.186541 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" podUID="c56971b8-db79-464a-bdaa-1a1e950f7c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 15:31:48.185134 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:48.184765 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pvhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 15:31:48.185134 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:48.184823 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" podUID="c56971b8-db79-464a-bdaa-1a1e950f7c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 15:31:49.184538 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:49.184495 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pvhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 15:31:49.184760 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:49.184567 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" podUID="c56971b8-db79-464a-bdaa-1a1e950f7c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 15:31:50.184973 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:50.184928 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pvhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 15:31:50.185486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:50.184995 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" podUID="c56971b8-db79-464a-bdaa-1a1e950f7c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 15:31:51.184585 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:51.184543 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pvhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 15:31:51.184785 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:51.184612 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" podUID="c56971b8-db79-464a-bdaa-1a1e950f7c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 15:31:52.185373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:52.185333 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pvhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 15:31:52.185858 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:52.185400 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" podUID="c56971b8-db79-464a-bdaa-1a1e950f7c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 15:31:53.185322 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:53.185264 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pvhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 15:31:53.185518 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:53.185342 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" podUID="c56971b8-db79-464a-bdaa-1a1e950f7c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 15:31:54.505784 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:54.505755 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:54.506761 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:54.506576 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:54.508158 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:54.508133 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" Nov 03 15:31:58.382153 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:58.382089 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:58.488305 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:58.488257 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:58.521823 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:58.521778 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:58.604968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:58.603789 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:58.955889 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:58.955832 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:59.085726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:59.083669 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:59.149726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:59.149000 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:59.238789 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:59.237895 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:59.404507 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:59.400633 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 15:31:59.404507 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:59.401395 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:31:59.493825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:59.493723 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:59.494493 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:59.494467 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:31:59.551739 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:59.535486 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Nov 03 15:31:59.551739 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:59.535864 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:31:59.726864 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:59.726798 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 15:31:59.731749 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:59.729157 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:31:59.935734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:31:59.927914 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:00.018332 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:00.018286 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Nov 03 15:32:00.046207 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:00.046173 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:32:00.234876 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:00.089870 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 03 15:32:00.236311 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:00.166870 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:00.251910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:00.203105 4290 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded" Nov 03 15:32:00.251910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:00.246115 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 15:32:00.296467 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:00.296439 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:32:00.297120 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:00.297103 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:32:00.366467 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:00.366437 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:32:00.421504 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:00.421454 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:00.548780 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:00.523506 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 15:32:00.550933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:00.550891 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:00.821828 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:00.778626 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:01.078434 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:01.078339 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Nov 03 15:32:01.242017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:01.241973 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:01.243272 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:01.243241 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:01.251797 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:01.251763 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:01.353229 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:01.353118 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:32:01.452555 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:01.452428 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 15:32:01.575748 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:01.559187 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 15:32:01.663362 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:01.663328 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:32:01.823303 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:01.823255 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 15:32:01.927918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:01.925992 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:32:01.950174 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:01.947758 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:32:01.999155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:01.998044 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:32:02.050140 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:02.050106 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:32:02.120344 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:02.120295 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:32:02.120545 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:02.120377 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:32:08.709842 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:32:08.709748 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:32:34.422529 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:34.422491 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd"] Nov 03 15:32:34.537005 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:34.536970 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7"] Nov 03 15:32:38.721410 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:32:38.721366 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:32:39.423235 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.423138 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" podUID="1bd8c668-f317-4c36-b64a-e4dab6a9cfe0" containerName="payload-processing" containerID="cri-o://62941ec3e17a96a6fcd81573d139d92d712669478163c630888d35d9d090e8c7" gracePeriod=25 Nov 03 15:32:39.537376 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.537329 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" podUID="ce976803-7a19-4e58-a3bc-e1132338b811" containerName="payload-pre-processing" containerID="cri-o://9106e69183de05b32b43795db0c4d813ee855320af884a1fe0bd2e8a1fedd686" gracePeriod=25 Nov 03 15:32:39.756769 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.756741 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" Nov 03 15:32:39.802718 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.802136 4290 generic.go:356] "Generic (PLEG): container finished" podID="1bd8c668-f317-4c36-b64a-e4dab6a9cfe0" containerID="62941ec3e17a96a6fcd81573d139d92d712669478163c630888d35d9d090e8c7" exitCode=0 Nov 03 15:32:39.802718 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.802452 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" event={"ID":"1bd8c668-f317-4c36-b64a-e4dab6a9cfe0","Type":"ContainerDied","Data":"62941ec3e17a96a6fcd81573d139d92d712669478163c630888d35d9d090e8c7"} Nov 03 15:32:39.802718 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.802504 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" event={"ID":"1bd8c668-f317-4c36-b64a-e4dab6a9cfe0","Type":"ContainerDied","Data":"3a0fb77b487a1af9477ed3a38682bbe6e602946f7e57df17335a2938b9c4f22a"} Nov 03 15:32:39.802718 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.802530 4290 scope.go:122] "RemoveContainer" containerID="62941ec3e17a96a6fcd81573d139d92d712669478163c630888d35d9d090e8c7" Nov 03 15:32:39.802718 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.802717 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd" Nov 03 15:32:39.807001 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.806971 4290 generic.go:356] "Generic (PLEG): container finished" podID="ce976803-7a19-4e58-a3bc-e1132338b811" containerID="9106e69183de05b32b43795db0c4d813ee855320af884a1fe0bd2e8a1fedd686" exitCode=0 Nov 03 15:32:39.807443 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.807021 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" event={"ID":"ce976803-7a19-4e58-a3bc-e1132338b811","Type":"ContainerDied","Data":"9106e69183de05b32b43795db0c4d813ee855320af884a1fe0bd2e8a1fedd686"} Nov 03 15:32:39.822895 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.822872 4290 scope.go:122] "RemoveContainer" containerID="62941ec3e17a96a6fcd81573d139d92d712669478163c630888d35d9d090e8c7" Nov 03 15:32:39.823261 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:32:39.823233 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62941ec3e17a96a6fcd81573d139d92d712669478163c630888d35d9d090e8c7\": container with ID starting with 62941ec3e17a96a6fcd81573d139d92d712669478163c630888d35d9d090e8c7 not found: ID does not exist" containerID="62941ec3e17a96a6fcd81573d139d92d712669478163c630888d35d9d090e8c7" Nov 03 15:32:39.823364 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.823274 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62941ec3e17a96a6fcd81573d139d92d712669478163c630888d35d9d090e8c7"} err="failed to get container status \"62941ec3e17a96a6fcd81573d139d92d712669478163c630888d35d9d090e8c7\": rpc error: code = NotFound desc = could not find container \"62941ec3e17a96a6fcd81573d139d92d712669478163c630888d35d9d090e8c7\": container with ID starting with 62941ec3e17a96a6fcd81573d139d92d712669478163c630888d35d9d090e8c7 not found: ID does not exist" Nov 03 15:32:39.870889 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.870849 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd8c668-f317-4c36-b64a-e4dab6a9cfe0-kube-api-access-wxf5x\" (UniqueName: \"kubernetes.io/projected/1bd8c668-f317-4c36-b64a-e4dab6a9cfe0-kube-api-access-wxf5x\") pod \"1bd8c668-f317-4c36-b64a-e4dab6a9cfe0\" (UID: \"1bd8c668-f317-4c36-b64a-e4dab6a9cfe0\") " Nov 03 15:32:39.871079 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.870915 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bd8c668-f317-4c36-b64a-e4dab6a9cfe0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd8c668-f317-4c36-b64a-e4dab6a9cfe0-plugins-config-volume\") pod \"1bd8c668-f317-4c36-b64a-e4dab6a9cfe0\" (UID: \"1bd8c668-f317-4c36-b64a-e4dab6a9cfe0\") " Nov 03 15:32:39.872831 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.872798 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd8c668-f317-4c36-b64a-e4dab6a9cfe0-plugins-config-volume" pod "1bd8c668-f317-4c36-b64a-e4dab6a9cfe0" (UID: "1bd8c668-f317-4c36-b64a-e4dab6a9cfe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:32:39.891776 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.876285 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd8c668-f317-4c36-b64a-e4dab6a9cfe0-kube-api-access-wxf5x" pod "1bd8c668-f317-4c36-b64a-e4dab6a9cfe0" (UID: "1bd8c668-f317-4c36-b64a-e4dab6a9cfe0"). InnerVolumeSpecName "kube-api-access-wxf5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:32:39.913760 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.913676 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" Nov 03 15:32:39.972773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.972668 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce976803-7a19-4e58-a3bc-e1132338b811-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce976803-7a19-4e58-a3bc-e1132338b811-plugins-config-volume\") pod \"ce976803-7a19-4e58-a3bc-e1132338b811\" (UID: \"ce976803-7a19-4e58-a3bc-e1132338b811\") " Nov 03 15:32:39.972954 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.972778 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce976803-7a19-4e58-a3bc-e1132338b811-kube-api-access-zgckx\" (UniqueName: \"kubernetes.io/projected/ce976803-7a19-4e58-a3bc-e1132338b811-kube-api-access-zgckx\") pod \"ce976803-7a19-4e58-a3bc-e1132338b811\" (UID: \"ce976803-7a19-4e58-a3bc-e1132338b811\") " Nov 03 15:32:39.973128 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.973099 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce976803-7a19-4e58-a3bc-e1132338b811-plugins-config-volume" pod "ce976803-7a19-4e58-a3bc-e1132338b811" (UID: "ce976803-7a19-4e58-a3bc-e1132338b811"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:32:39.973503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.973487 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce976803-7a19-4e58-a3bc-e1132338b811-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:39.973582 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.973511 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxf5x\" (UniqueName: \"kubernetes.io/projected/1bd8c668-f317-4c36-b64a-e4dab6a9cfe0-kube-api-access-wxf5x\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:39.973582 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.973522 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd8c668-f317-4c36-b64a-e4dab6a9cfe0-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:39.975035 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:39.975008 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce976803-7a19-4e58-a3bc-e1132338b811-kube-api-access-zgckx" pod "ce976803-7a19-4e58-a3bc-e1132338b811" (UID: "ce976803-7a19-4e58-a3bc-e1132338b811"). InnerVolumeSpecName "kube-api-access-zgckx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:32:40.081496 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:40.081453 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgckx\" (UniqueName: \"kubernetes.io/projected/ce976803-7a19-4e58-a3bc-e1132338b811-kube-api-access-zgckx\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:40.250615 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:40.250508 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd"] Nov 03 15:32:40.255986 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:40.255956 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-47b989b3-58d8df68c-9b8cd"] Nov 03 15:32:40.719723 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:40.718902 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bd8c668-f317-4c36-b64a-e4dab6a9cfe0" path="/var/lib/kubelet/pods/1bd8c668-f317-4c36-b64a-e4dab6a9cfe0/volumes" Nov 03 15:32:40.876845 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:40.867498 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" Nov 03 15:32:40.876845 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:40.874900 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7" event={"ID":"ce976803-7a19-4e58-a3bc-e1132338b811","Type":"ContainerDied","Data":"08dcdad6c3a31898d22718c7278c1c12b5ff248d51f52d9313c97605d8494c7b"} Nov 03 15:32:40.876845 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:40.875239 4290 scope.go:122] "RemoveContainer" containerID="9106e69183de05b32b43795db0c4d813ee855320af884a1fe0bd2e8a1fedd686" Nov 03 15:32:41.060002 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:41.059901 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7"] Nov 03 15:32:41.105105 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:41.105067 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-47b989b3-75f65c46b7-8hwx7"] Nov 03 15:32:41.610239 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:41.610205 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk"] Nov 03 15:32:41.726291 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:41.726255 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf"] Nov 03 15:32:42.699128 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:42.699091 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce976803-7a19-4e58-a3bc-e1132338b811" path="/var/lib/kubelet/pods/ce976803-7a19-4e58-a3bc-e1132338b811/volumes" Nov 03 15:32:46.610730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.610658 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" podUID="c1bbe6d2-f247-49f2-83ff-097721fcacbd" containerName="payload-processing" containerID="cri-o://e06ac7f9a91c7c671fb9c2f0791f29615c7c55d25bc1dc90fe58467e5920f63a" gracePeriod=25 Nov 03 15:32:46.727373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.727319 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" podUID="33764de5-4ee3-4c10-b63c-17b701bca3bd" containerName="payload-pre-processing" containerID="cri-o://f0824e47100351847627ec42388ee95044be50e6bc997da67f200fecd153fed8" gracePeriod=25 Nov 03 15:32:46.924218 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.923895 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" Nov 03 15:32:46.944758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.942905 4290 generic.go:356] "Generic (PLEG): container finished" podID="33764de5-4ee3-4c10-b63c-17b701bca3bd" containerID="f0824e47100351847627ec42388ee95044be50e6bc997da67f200fecd153fed8" exitCode=0 Nov 03 15:32:46.944758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.942987 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" event={"ID":"33764de5-4ee3-4c10-b63c-17b701bca3bd","Type":"ContainerDied","Data":"f0824e47100351847627ec42388ee95044be50e6bc997da67f200fecd153fed8"} Nov 03 15:32:46.944989 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.944800 4290 generic.go:356] "Generic (PLEG): container finished" podID="c1bbe6d2-f247-49f2-83ff-097721fcacbd" containerID="e06ac7f9a91c7c671fb9c2f0791f29615c7c55d25bc1dc90fe58467e5920f63a" exitCode=0 Nov 03 15:32:46.944989 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.944952 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" event={"ID":"c1bbe6d2-f247-49f2-83ff-097721fcacbd","Type":"ContainerDied","Data":"e06ac7f9a91c7c671fb9c2f0791f29615c7c55d25bc1dc90fe58467e5920f63a"} Nov 03 15:32:46.945107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.944993 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" event={"ID":"c1bbe6d2-f247-49f2-83ff-097721fcacbd","Type":"ContainerDied","Data":"53024a32276a7264eb05aa944abfa5c9315367dd472c7a41f1a6d474ff5e6a92"} Nov 03 15:32:46.945107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.945016 4290 scope.go:122] "RemoveContainer" containerID="e06ac7f9a91c7c671fb9c2f0791f29615c7c55d25bc1dc90fe58467e5920f63a" Nov 03 15:32:46.945214 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.945143 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk" Nov 03 15:32:46.960688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.960663 4290 scope.go:122] "RemoveContainer" containerID="e06ac7f9a91c7c671fb9c2f0791f29615c7c55d25bc1dc90fe58467e5920f63a" Nov 03 15:32:46.961078 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:32:46.961050 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06ac7f9a91c7c671fb9c2f0791f29615c7c55d25bc1dc90fe58467e5920f63a\": container with ID starting with e06ac7f9a91c7c671fb9c2f0791f29615c7c55d25bc1dc90fe58467e5920f63a not found: ID does not exist" containerID="e06ac7f9a91c7c671fb9c2f0791f29615c7c55d25bc1dc90fe58467e5920f63a" Nov 03 15:32:46.961172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.961091 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06ac7f9a91c7c671fb9c2f0791f29615c7c55d25bc1dc90fe58467e5920f63a"} err="failed to get container status \"e06ac7f9a91c7c671fb9c2f0791f29615c7c55d25bc1dc90fe58467e5920f63a\": rpc error: code = NotFound desc = could not find container \"e06ac7f9a91c7c671fb9c2f0791f29615c7c55d25bc1dc90fe58467e5920f63a\": container with ID starting with e06ac7f9a91c7c671fb9c2f0791f29615c7c55d25bc1dc90fe58467e5920f63a not found: ID does not exist" Nov 03 15:32:46.961574 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.961546 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1bbe6d2-f247-49f2-83ff-097721fcacbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1bbe6d2-f247-49f2-83ff-097721fcacbd-plugins-config-volume\") pod \"c1bbe6d2-f247-49f2-83ff-097721fcacbd\" (UID: \"c1bbe6d2-f247-49f2-83ff-097721fcacbd\") " Nov 03 15:32:46.961668 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.961637 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1bbe6d2-f247-49f2-83ff-097721fcacbd-kube-api-access-qlwjn\" (UniqueName: \"kubernetes.io/projected/c1bbe6d2-f247-49f2-83ff-097721fcacbd-kube-api-access-qlwjn\") pod \"c1bbe6d2-f247-49f2-83ff-097721fcacbd\" (UID: \"c1bbe6d2-f247-49f2-83ff-097721fcacbd\") " Nov 03 15:32:46.962404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.962376 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1bbe6d2-f247-49f2-83ff-097721fcacbd-plugins-config-volume" pod "c1bbe6d2-f247-49f2-83ff-097721fcacbd" (UID: "c1bbe6d2-f247-49f2-83ff-097721fcacbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:32:46.962795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.962774 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1bbe6d2-f247-49f2-83ff-097721fcacbd-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:46.965156 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:46.965127 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1bbe6d2-f247-49f2-83ff-097721fcacbd-kube-api-access-qlwjn" pod "c1bbe6d2-f247-49f2-83ff-097721fcacbd" (UID: "c1bbe6d2-f247-49f2-83ff-097721fcacbd"). InnerVolumeSpecName "kube-api-access-qlwjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:32:47.030823 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:47.030798 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" Nov 03 15:32:47.063549 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:47.063513 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33764de5-4ee3-4c10-b63c-17b701bca3bd-kube-api-access-hz4qp\" (UniqueName: \"kubernetes.io/projected/33764de5-4ee3-4c10-b63c-17b701bca3bd-kube-api-access-hz4qp\") pod \"33764de5-4ee3-4c10-b63c-17b701bca3bd\" (UID: \"33764de5-4ee3-4c10-b63c-17b701bca3bd\") " Nov 03 15:32:47.063783 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:47.063679 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33764de5-4ee3-4c10-b63c-17b701bca3bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33764de5-4ee3-4c10-b63c-17b701bca3bd-plugins-config-volume\") pod \"33764de5-4ee3-4c10-b63c-17b701bca3bd\" (UID: \"33764de5-4ee3-4c10-b63c-17b701bca3bd\") " Nov 03 15:32:47.064571 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:47.064536 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33764de5-4ee3-4c10-b63c-17b701bca3bd-plugins-config-volume" pod "33764de5-4ee3-4c10-b63c-17b701bca3bd" (UID: "33764de5-4ee3-4c10-b63c-17b701bca3bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:32:47.064765 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:47.064739 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33764de5-4ee3-4c10-b63c-17b701bca3bd-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:47.064844 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:47.064773 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlwjn\" (UniqueName: \"kubernetes.io/projected/c1bbe6d2-f247-49f2-83ff-097721fcacbd-kube-api-access-qlwjn\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:47.066307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:47.066277 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33764de5-4ee3-4c10-b63c-17b701bca3bd-kube-api-access-hz4qp" pod "33764de5-4ee3-4c10-b63c-17b701bca3bd" (UID: "33764de5-4ee3-4c10-b63c-17b701bca3bd"). InnerVolumeSpecName "kube-api-access-hz4qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:32:47.167037 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:47.166965 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz4qp\" (UniqueName: \"kubernetes.io/projected/33764de5-4ee3-4c10-b63c-17b701bca3bd-kube-api-access-hz4qp\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:47.373837 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:47.373806 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk"] Nov 03 15:32:47.379395 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:47.379365 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d785c53b-74597b6b5-c5gmk"] Nov 03 15:32:47.955061 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:47.954980 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" Nov 03 15:32:47.955061 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:47.955016 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf" event={"ID":"33764de5-4ee3-4c10-b63c-17b701bca3bd","Type":"ContainerDied","Data":"c5e83c22e8d83908e03f54a6113d46d3a61d234dcf119c2c5e8fe9df6fa24ba7"} Nov 03 15:32:47.955590 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:47.955091 4290 scope.go:122] "RemoveContainer" containerID="f0824e47100351847627ec42388ee95044be50e6bc997da67f200fecd153fed8" Nov 03 15:32:48.123440 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:48.123409 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf"] Nov 03 15:32:48.175668 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:48.175631 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d785c53b-66c8769c99-rftwf"] Nov 03 15:32:48.680532 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:48.680497 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33764de5-4ee3-4c10-b63c-17b701bca3bd" path="/var/lib/kubelet/pods/33764de5-4ee3-4c10-b63c-17b701bca3bd/volumes" Nov 03 15:32:48.681063 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:48.681043 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1bbe6d2-f247-49f2-83ff-097721fcacbd" path="/var/lib/kubelet/pods/c1bbe6d2-f247-49f2-83ff-097721fcacbd/volumes" Nov 03 15:32:57.711720 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:57.711625 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl"] Nov 03 15:32:57.825804 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:32:57.825765 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7"] Nov 03 15:33:02.712221 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:02.712170 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" podUID="2417d881-270c-4ce5-a27f-fa8737bd56eb" containerName="payload-processing" containerID="cri-o://7a67e4eb525d82f3cac5f890ff241d5075e86d8c5bb3ff5244cec54c7b8c7044" gracePeriod=25 Nov 03 15:33:02.826179 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:02.826141 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" podUID="a9a549ce-c7d4-4c27-96a1-2a030683b871" containerName="payload-pre-processing" containerID="cri-o://26ce160af85ab86a97429a4d2e9ba641c2149b9f3aa4af2e395fa951a0d19551" gracePeriod=25 Nov 03 15:33:03.031756 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.031562 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" Nov 03 15:33:03.110953 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.110921 4290 generic.go:356] "Generic (PLEG): container finished" podID="2417d881-270c-4ce5-a27f-fa8737bd56eb" containerID="7a67e4eb525d82f3cac5f890ff241d5075e86d8c5bb3ff5244cec54c7b8c7044" exitCode=0 Nov 03 15:33:03.111149 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.111084 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" event={"ID":"2417d881-270c-4ce5-a27f-fa8737bd56eb","Type":"ContainerDied","Data":"7a67e4eb525d82f3cac5f890ff241d5075e86d8c5bb3ff5244cec54c7b8c7044"} Nov 03 15:33:03.111149 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.111137 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" event={"ID":"2417d881-270c-4ce5-a27f-fa8737bd56eb","Type":"ContainerDied","Data":"3cbee401729b2961d3c68afa8b70df2f59b91812b54f15efcc9301588841961d"} Nov 03 15:33:03.111272 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.111164 4290 scope.go:122] "RemoveContainer" containerID="7a67e4eb525d82f3cac5f890ff241d5075e86d8c5bb3ff5244cec54c7b8c7044" Nov 03 15:33:03.111326 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.111299 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl" Nov 03 15:33:03.114758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.114689 4290 generic.go:356] "Generic (PLEG): container finished" podID="a9a549ce-c7d4-4c27-96a1-2a030683b871" containerID="26ce160af85ab86a97429a4d2e9ba641c2149b9f3aa4af2e395fa951a0d19551" exitCode=0 Nov 03 15:33:03.114990 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.114940 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" event={"ID":"a9a549ce-c7d4-4c27-96a1-2a030683b871","Type":"ContainerDied","Data":"26ce160af85ab86a97429a4d2e9ba641c2149b9f3aa4af2e395fa951a0d19551"} Nov 03 15:33:03.136182 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.136147 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2417d881-270c-4ce5-a27f-fa8737bd56eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2417d881-270c-4ce5-a27f-fa8737bd56eb-plugins-config-volume\") pod \"2417d881-270c-4ce5-a27f-fa8737bd56eb\" (UID: \"2417d881-270c-4ce5-a27f-fa8737bd56eb\") " Nov 03 15:33:03.136301 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.136286 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2417d881-270c-4ce5-a27f-fa8737bd56eb-kube-api-access-65hm4\" (UniqueName: \"kubernetes.io/projected/2417d881-270c-4ce5-a27f-fa8737bd56eb-kube-api-access-65hm4\") pod \"2417d881-270c-4ce5-a27f-fa8737bd56eb\" (UID: \"2417d881-270c-4ce5-a27f-fa8737bd56eb\") " Nov 03 15:33:03.136521 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.136492 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2417d881-270c-4ce5-a27f-fa8737bd56eb-plugins-config-volume" pod "2417d881-270c-4ce5-a27f-fa8737bd56eb" (UID: "2417d881-270c-4ce5-a27f-fa8737bd56eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:33:03.137022 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.137002 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2417d881-270c-4ce5-a27f-fa8737bd56eb-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:03.138568 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.138495 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2417d881-270c-4ce5-a27f-fa8737bd56eb-kube-api-access-65hm4" pod "2417d881-270c-4ce5-a27f-fa8737bd56eb" (UID: "2417d881-270c-4ce5-a27f-fa8737bd56eb"). InnerVolumeSpecName "kube-api-access-65hm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:33:03.144779 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.144756 4290 scope.go:122] "RemoveContainer" containerID="7a67e4eb525d82f3cac5f890ff241d5075e86d8c5bb3ff5244cec54c7b8c7044" Nov 03 15:33:03.145074 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:33:03.145054 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a67e4eb525d82f3cac5f890ff241d5075e86d8c5bb3ff5244cec54c7b8c7044\": container with ID starting with 7a67e4eb525d82f3cac5f890ff241d5075e86d8c5bb3ff5244cec54c7b8c7044 not found: ID does not exist" containerID="7a67e4eb525d82f3cac5f890ff241d5075e86d8c5bb3ff5244cec54c7b8c7044" Nov 03 15:33:03.145150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.145081 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a67e4eb525d82f3cac5f890ff241d5075e86d8c5bb3ff5244cec54c7b8c7044"} err="failed to get container status \"7a67e4eb525d82f3cac5f890ff241d5075e86d8c5bb3ff5244cec54c7b8c7044\": rpc error: code = NotFound desc = could not find container \"7a67e4eb525d82f3cac5f890ff241d5075e86d8c5bb3ff5244cec54c7b8c7044\": container with ID starting with 7a67e4eb525d82f3cac5f890ff241d5075e86d8c5bb3ff5244cec54c7b8c7044 not found: ID does not exist" Nov 03 15:33:03.154039 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.153178 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" Nov 03 15:33:03.238159 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.238122 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a549ce-c7d4-4c27-96a1-2a030683b871-kube-api-access-k2kck\" (UniqueName: \"kubernetes.io/projected/a9a549ce-c7d4-4c27-96a1-2a030683b871-kube-api-access-k2kck\") pod \"a9a549ce-c7d4-4c27-96a1-2a030683b871\" (UID: \"a9a549ce-c7d4-4c27-96a1-2a030683b871\") " Nov 03 15:33:03.238334 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.238262 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a549ce-c7d4-4c27-96a1-2a030683b871-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a549ce-c7d4-4c27-96a1-2a030683b871-plugins-config-volume\") pod \"a9a549ce-c7d4-4c27-96a1-2a030683b871\" (UID: \"a9a549ce-c7d4-4c27-96a1-2a030683b871\") " Nov 03 15:33:03.240377 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.239310 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a549ce-c7d4-4c27-96a1-2a030683b871-plugins-config-volume" pod "a9a549ce-c7d4-4c27-96a1-2a030683b871" (UID: "a9a549ce-c7d4-4c27-96a1-2a030683b871"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:33:03.241866 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.241178 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a549ce-c7d4-4c27-96a1-2a030683b871-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:03.241866 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.241214 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65hm4\" (UniqueName: \"kubernetes.io/projected/2417d881-270c-4ce5-a27f-fa8737bd56eb-kube-api-access-65hm4\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:03.255089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.255053 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a549ce-c7d4-4c27-96a1-2a030683b871-kube-api-access-k2kck" pod "a9a549ce-c7d4-4c27-96a1-2a030683b871" (UID: "a9a549ce-c7d4-4c27-96a1-2a030683b871"). InnerVolumeSpecName "kube-api-access-k2kck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:33:03.342716 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.342675 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2kck\" (UniqueName: \"kubernetes.io/projected/a9a549ce-c7d4-4c27-96a1-2a030683b871-kube-api-access-k2kck\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:03.537410 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.537333 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl"] Nov 03 15:33:03.543186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:03.543157 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cceaf27f-55994d64f7-6b7tl"] Nov 03 15:33:04.130884 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:04.130848 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" Nov 03 15:33:04.131320 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:04.130843 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7" event={"ID":"a9a549ce-c7d4-4c27-96a1-2a030683b871","Type":"ContainerDied","Data":"fa49503b212bf84c55c6c3af44406ddec3ba14f4f284dc7cd9d361012f1cd870"} Nov 03 15:33:04.131320 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:04.130992 4290 scope.go:122] "RemoveContainer" containerID="26ce160af85ab86a97429a4d2e9ba641c2149b9f3aa4af2e395fa951a0d19551" Nov 03 15:33:04.300149 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:04.300112 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7"] Nov 03 15:33:04.349988 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:04.349954 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cceaf27f-654dc5d4d7-xmwz7"] Nov 03 15:33:04.684795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:04.684760 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2417d881-270c-4ce5-a27f-fa8737bd56eb" path="/var/lib/kubelet/pods/2417d881-270c-4ce5-a27f-fa8737bd56eb/volumes" Nov 03 15:33:04.685443 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:04.685425 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a549ce-c7d4-4c27-96a1-2a030683b871" path="/var/lib/kubelet/pods/a9a549ce-c7d4-4c27-96a1-2a030683b871/volumes" Nov 03 15:33:08.709847 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:33:08.709760 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:33:12.925091 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:12.925059 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:33:12.925764 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:12.925747 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:33:12.928367 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:12.928349 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:33:12.928649 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:33:12.928636 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:33:38.713492 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:33:38.713447 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:34:08.703656 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:34:08.703570 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:34:35.060221 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:34:35.060139 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh"] Nov 03 15:34:35.107065 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:34:35.107035 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2615d260-hrvnh"] Nov 03 15:34:36.725299 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:34:36.725246 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03d86200-4c4d-4be5-a003-4dee8e1f974f" path="/var/lib/kubelet/pods/03d86200-4c4d-4be5-a003-4dee8e1f974f/volumes" Nov 03 15:34:38.705438 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:34:38.705399 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:35:05.058307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:05.058275 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9"] Nov 03 15:35:05.107762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:05.107724 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e0fe64-xzzh9"] Nov 03 15:35:06.720086 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:06.720050 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ec8782-7966-4c2d-929a-197fb81ad397" path="/var/lib/kubelet/pods/76ec8782-7966-4c2d-929a-197fb81ad397/volumes" Nov 03 15:35:08.713962 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:35:08.713869 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:35:31.052374 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:31.052340 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs"] Nov 03 15:35:31.101884 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:31.101852 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ab54ac5-lbrfs"] Nov 03 15:35:32.680144 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:32.680114 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d955c5f2-5757-4b5e-9667-9e3db67795bf" path="/var/lib/kubelet/pods/d955c5f2-5757-4b5e-9667-9e3db67795bf/volumes" Nov 03 15:35:38.705469 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:35:38.705433 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:35:39.918689 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.918621 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl"] Nov 03 15:35:39.925776 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.925665 4290 cpu_manager.go:414] "containerMap: removing container" podUID="ce976803-7a19-4e58-a3bc-e1132338b811" containerName="payload-pre-processing" Nov 03 15:35:39.925959 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.925877 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce976803-7a19-4e58-a3bc-e1132338b811" containerName="payload-pre-processing" Nov 03 15:35:39.925959 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.925915 4290 cpu_manager.go:414] "containerMap: removing container" podUID="c1bbe6d2-f247-49f2-83ff-097721fcacbd" containerName="payload-processing" Nov 03 15:35:39.925959 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.925923 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1bbe6d2-f247-49f2-83ff-097721fcacbd" containerName="payload-processing" Nov 03 15:35:39.925959 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.925951 4290 cpu_manager.go:414] "containerMap: removing container" podUID="1bd8c668-f317-4c36-b64a-e4dab6a9cfe0" containerName="payload-processing" Nov 03 15:35:39.926186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.925961 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd8c668-f317-4c36-b64a-e4dab6a9cfe0" containerName="payload-processing" Nov 03 15:35:39.926186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.926005 4290 cpu_manager.go:414] "containerMap: removing container" podUID="a9a549ce-c7d4-4c27-96a1-2a030683b871" containerName="payload-pre-processing" Nov 03 15:35:39.926186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.926013 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a549ce-c7d4-4c27-96a1-2a030683b871" containerName="payload-pre-processing" Nov 03 15:35:39.926186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.926039 4290 cpu_manager.go:414] "containerMap: removing container" podUID="33764de5-4ee3-4c10-b63c-17b701bca3bd" containerName="payload-pre-processing" Nov 03 15:35:39.926186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.926046 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33764de5-4ee3-4c10-b63c-17b701bca3bd" containerName="payload-pre-processing" Nov 03 15:35:39.926186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.926073 4290 cpu_manager.go:414] "containerMap: removing container" podUID="2417d881-270c-4ce5-a27f-fa8737bd56eb" containerName="payload-processing" Nov 03 15:35:39.926186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.926080 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2417d881-270c-4ce5-a27f-fa8737bd56eb" containerName="payload-processing" Nov 03 15:35:39.926513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.926303 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce976803-7a19-4e58-a3bc-e1132338b811" containerName="payload-pre-processing" Nov 03 15:35:39.926513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.926320 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1bbe6d2-f247-49f2-83ff-097721fcacbd" containerName="payload-processing" Nov 03 15:35:39.926513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.926334 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd8c668-f317-4c36-b64a-e4dab6a9cfe0" containerName="payload-processing" Nov 03 15:35:39.926513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.926351 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a549ce-c7d4-4c27-96a1-2a030683b871" containerName="payload-pre-processing" Nov 03 15:35:39.926513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.926363 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="33764de5-4ee3-4c10-b63c-17b701bca3bd" containerName="payload-pre-processing" Nov 03 15:35:39.926513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.926373 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="2417d881-270c-4ce5-a27f-fa8737bd56eb" containerName="payload-processing" Nov 03 15:35:39.933194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.933170 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" Nov 03 15:35:39.978134 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:39.978102 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl"] Nov 03 15:35:40.021830 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:40.021794 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5cad072f\"" Nov 03 15:35:40.022514 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:40.022470 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5cad072f-dockercfg-6sv7w\"" Nov 03 15:35:40.081430 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:40.079889 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmqbc\" (UniqueName: \"kubernetes.io/projected/e8ba51c9-602b-47a6-bca9-5f9fa7547276-kube-api-access-mmqbc\") pod \"payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl\" (UID: \"e8ba51c9-602b-47a6-bca9-5f9fa7547276\") " pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" Nov 03 15:35:40.081430 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:40.080042 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ba51c9-602b-47a6-bca9-5f9fa7547276-plugins-config-volume\") pod \"payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl\" (UID: \"e8ba51c9-602b-47a6-bca9-5f9fa7547276\") " pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" Nov 03 15:35:40.181677 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:40.181578 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmqbc\" (UniqueName: \"kubernetes.io/projected/e8ba51c9-602b-47a6-bca9-5f9fa7547276-kube-api-access-mmqbc\") pod \"payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl\" (UID: \"e8ba51c9-602b-47a6-bca9-5f9fa7547276\") " pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" Nov 03 15:35:40.181864 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:40.181725 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ba51c9-602b-47a6-bca9-5f9fa7547276-plugins-config-volume\") pod \"payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl\" (UID: \"e8ba51c9-602b-47a6-bca9-5f9fa7547276\") " pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" Nov 03 15:35:40.182464 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:40.182445 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ba51c9-602b-47a6-bca9-5f9fa7547276-plugins-config-volume\") pod \"payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl\" (UID: \"e8ba51c9-602b-47a6-bca9-5f9fa7547276\") " pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" Nov 03 15:35:40.243828 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:40.243799 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmqbc\" (UniqueName: \"kubernetes.io/projected/e8ba51c9-602b-47a6-bca9-5f9fa7547276-kube-api-access-mmqbc\") pod \"payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl\" (UID: \"e8ba51c9-602b-47a6-bca9-5f9fa7547276\") " pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" Nov 03 15:35:40.267923 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:40.267889 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" Nov 03 15:35:40.754253 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:35:40.754217 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ba51c9_602b_47a6_bca9_5f9fa7547276.slice/crio-015a1452feb5481ea737b55af4d04a5ce39f2af1a6ecaf81057f2bb54139fc14 WatchSource:0}: Error finding container 015a1452feb5481ea737b55af4d04a5ce39f2af1a6ecaf81057f2bb54139fc14: Status 404 returned error can't find the container with id 015a1452feb5481ea737b55af4d04a5ce39f2af1a6ecaf81057f2bb54139fc14 Nov 03 15:35:40.754533 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:40.754511 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl"] Nov 03 15:35:40.894439 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:40.894408 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" event={"ID":"e8ba51c9-602b-47a6-bca9-5f9fa7547276","Type":"ContainerStarted","Data":"6fc19d5d046165da0348bda2470167353008907a464f6b85d4979119b67c8db3"} Nov 03 15:35:40.894620 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:40.894465 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" event={"ID":"e8ba51c9-602b-47a6-bca9-5f9fa7547276","Type":"ContainerStarted","Data":"015a1452feb5481ea737b55af4d04a5ce39f2af1a6ecaf81057f2bb54139fc14"} Nov 03 15:35:40.894620 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:40.894495 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" Nov 03 15:35:41.012770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:41.012619 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" podStartSLOduration=2.012604479 podStartE2EDuration="2.012604479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:35:41.009563224 +0000 UTC m=+1648.440521031" watchObservedRunningTime="2026-11-03 15:35:41.012604479 +0000 UTC m=+1648.443562285" Nov 03 15:35:51.919046 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:35:51.919008 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" Nov 03 15:36:08.704797 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:36:08.704678 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:36:16.936774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:16.936724 4290 scope.go:122] "RemoveContainer" containerID="ef956686ce6751053f59ff4d560d09bae78067e78434603e2d13155ea56e6a8d" Nov 03 15:36:16.948166 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:16.948143 4290 scope.go:122] "RemoveContainer" containerID="3ff05e8de2e77f92b702f8252e21cacfa03ee9d5c4e5df7f0e1778d904d7cbab" Nov 03 15:36:35.048134 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:35.048092 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk"] Nov 03 15:36:35.048633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:35.048436 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" podUID="2af12a1b-e8c5-418d-b61d-f9928a8213b6" containerName="istio-proxy" containerID="cri-o://0dbabd2c8da12d7dfbb44b4b22b2eea6ac61051d976d17c0c2eb55a87ec513fe" gracePeriod=30 Nov 03 15:36:35.349570 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:35.349472 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4"] Nov 03 15:36:35.350894 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:35.350834 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" podUID="28b9fcb2-e588-4d76-909c-244704b9a658" containerName="istio-proxy" containerID="cri-o://6f4f017f86a40b69b4bdc4098169dea614ce8fcabd461e3f58db9cb7daa0a122" gracePeriod=30 Nov 03 15:36:38.728304 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:36:38.728262 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:36:39.509465 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:39.509427 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:36:39.509629 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:39.509500 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" podUID="2af12a1b-e8c5-418d-b61d-f9928a8213b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:36:40.471290 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.471054 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:36:40.510873 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.510841 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-envoy\") pod \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " Nov 03 15:36:40.511076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.510897 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2af12a1b-e8c5-418d-b61d-f9928a8213b6-kube-api-access-qmlrl\" (UniqueName: \"kubernetes.io/projected/2af12a1b-e8c5-418d-b61d-f9928a8213b6-kube-api-access-qmlrl\") pod \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " Nov 03 15:36:40.511076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.510943 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-podinfo\") pod \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " Nov 03 15:36:40.511076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.510977 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-workload-socket\") pod \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " Nov 03 15:36:40.511076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.511003 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-credential-socket\") pod \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " Nov 03 15:36:40.511076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.511056 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-token\" (UniqueName: \"kubernetes.io/projected/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-token\") pod \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " Nov 03 15:36:40.511370 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.511132 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-workload-certs\") pod \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " Nov 03 15:36:40.511370 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.511277 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istiod-ca-cert\") pod \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " Nov 03 15:36:40.511370 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.511300 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-data\") pod \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\" (UID: \"2af12a1b-e8c5-418d-b61d-f9928a8213b6\") " Nov 03 15:36:40.512238 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.511831 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-workload-socket" pod "2af12a1b-e8c5-418d-b61d-f9928a8213b6" (UID: "2af12a1b-e8c5-418d-b61d-f9928a8213b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:40.512489 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.512460 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-workload-certs" pod "2af12a1b-e8c5-418d-b61d-f9928a8213b6" (UID: "2af12a1b-e8c5-418d-b61d-f9928a8213b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:40.512572 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.512525 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istiod-ca-cert" pod "2af12a1b-e8c5-418d-b61d-f9928a8213b6" (UID: "2af12a1b-e8c5-418d-b61d-f9928a8213b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:36:40.512639 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.512592 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-data" pod "2af12a1b-e8c5-418d-b61d-f9928a8213b6" (UID: "2af12a1b-e8c5-418d-b61d-f9928a8213b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:40.512734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.512676 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-credential-socket" pod "2af12a1b-e8c5-418d-b61d-f9928a8213b6" (UID: "2af12a1b-e8c5-418d-b61d-f9928a8213b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:40.516777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.514239 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-token" pod "2af12a1b-e8c5-418d-b61d-f9928a8213b6" (UID: "2af12a1b-e8c5-418d-b61d-f9928a8213b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:36:40.516777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.514849 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-envoy" pod "2af12a1b-e8c5-418d-b61d-f9928a8213b6" (UID: "2af12a1b-e8c5-418d-b61d-f9928a8213b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:40.516777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.515355 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-podinfo" pod "2af12a1b-e8c5-418d-b61d-f9928a8213b6" (UID: "2af12a1b-e8c5-418d-b61d-f9928a8213b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:36:40.516777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.516137 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af12a1b-e8c5-418d-b61d-f9928a8213b6-kube-api-access-qmlrl" pod "2af12a1b-e8c5-418d-b61d-f9928a8213b6" (UID: "2af12a1b-e8c5-418d-b61d-f9928a8213b6"). InnerVolumeSpecName "kube-api-access-qmlrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:36:40.615254 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.615211 4290 generic.go:356] "Generic (PLEG): container finished" podID="28b9fcb2-e588-4d76-909c-244704b9a658" containerID="6f4f017f86a40b69b4bdc4098169dea614ce8fcabd461e3f58db9cb7daa0a122" exitCode=0 Nov 03 15:36:40.615413 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.615396 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" event={"ID":"28b9fcb2-e588-4d76-909c-244704b9a658","Type":"ContainerDied","Data":"6f4f017f86a40b69b4bdc4098169dea614ce8fcabd461e3f58db9cb7daa0a122"} Nov 03 15:36:40.617814 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.617532 4290 generic.go:356] "Generic (PLEG): container finished" podID="2af12a1b-e8c5-418d-b61d-f9928a8213b6" containerID="0dbabd2c8da12d7dfbb44b4b22b2eea6ac61051d976d17c0c2eb55a87ec513fe" exitCode=0 Nov 03 15:36:40.618139 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.617715 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" event={"ID":"2af12a1b-e8c5-418d-b61d-f9928a8213b6","Type":"ContainerDied","Data":"0dbabd2c8da12d7dfbb44b4b22b2eea6ac61051d976d17c0c2eb55a87ec513fe"} Nov 03 15:36:40.618372 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.618351 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" event={"ID":"2af12a1b-e8c5-418d-b61d-f9928a8213b6","Type":"ContainerDied","Data":"c5d99da0d926134bde278cee7bc8e1ab2c4500fbae19c8a47efec19e244f5926"} Nov 03 15:36:40.618509 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.618493 4290 scope.go:122] "RemoveContainer" containerID="0dbabd2c8da12d7dfbb44b4b22b2eea6ac61051d976d17c0c2eb55a87ec513fe" Nov 03 15:36:40.618678 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.617803 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk" Nov 03 15:36:40.620286 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.620255 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.620286 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.620279 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.620672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.620308 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.620672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.620321 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.620672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.620335 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.620672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.620347 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmlrl\" (UniqueName: \"kubernetes.io/projected/2af12a1b-e8c5-418d-b61d-f9928a8213b6-kube-api-access-qmlrl\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.620672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.620366 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af12a1b-e8c5-418d-b61d-f9928a8213b6-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.620672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.620378 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.620672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.620390 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af12a1b-e8c5-418d-b61d-f9928a8213b6-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.711330 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.710744 4290 scope.go:122] "RemoveContainer" containerID="0dbabd2c8da12d7dfbb44b4b22b2eea6ac61051d976d17c0c2eb55a87ec513fe" Nov 03 15:36:40.712818 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:36:40.712322 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbabd2c8da12d7dfbb44b4b22b2eea6ac61051d976d17c0c2eb55a87ec513fe\": container with ID starting with 0dbabd2c8da12d7dfbb44b4b22b2eea6ac61051d976d17c0c2eb55a87ec513fe not found: ID does not exist" containerID="0dbabd2c8da12d7dfbb44b4b22b2eea6ac61051d976d17c0c2eb55a87ec513fe" Nov 03 15:36:40.712818 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.712370 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbabd2c8da12d7dfbb44b4b22b2eea6ac61051d976d17c0c2eb55a87ec513fe"} err="failed to get container status \"0dbabd2c8da12d7dfbb44b4b22b2eea6ac61051d976d17c0c2eb55a87ec513fe\": rpc error: code = NotFound desc = could not find container \"0dbabd2c8da12d7dfbb44b4b22b2eea6ac61051d976d17c0c2eb55a87ec513fe\": container with ID starting with 0dbabd2c8da12d7dfbb44b4b22b2eea6ac61051d976d17c0c2eb55a87ec513fe not found: ID does not exist" Nov 03 15:36:40.728633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.728575 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:36:40.793956 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.793927 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk"] Nov 03 15:36:40.827571 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.827528 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-m6vgk"] Nov 03 15:36:40.827896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.827873 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-istio-envoy\") pod \"28b9fcb2-e588-4d76-909c-244704b9a658\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " Nov 03 15:36:40.827996 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.827943 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b9fcb2-e588-4d76-909c-244704b9a658-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b9fcb2-e588-4d76-909c-244704b9a658-istiod-ca-cert\") pod \"28b9fcb2-e588-4d76-909c-244704b9a658\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " Nov 03 15:36:40.828073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.828044 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-istio-data\") pod \"28b9fcb2-e588-4d76-909c-244704b9a658\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " Nov 03 15:36:40.828073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.828067 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-workload-socket\") pod \"28b9fcb2-e588-4d76-909c-244704b9a658\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " Nov 03 15:36:40.828185 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.828103 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b9fcb2-e588-4d76-909c-244704b9a658-istio-token\" (UniqueName: \"kubernetes.io/projected/28b9fcb2-e588-4d76-909c-244704b9a658-istio-token\") pod \"28b9fcb2-e588-4d76-909c-244704b9a658\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " Nov 03 15:36:40.828185 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.828135 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-workload-certs\") pod \"28b9fcb2-e588-4d76-909c-244704b9a658\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " Nov 03 15:36:40.828290 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.828193 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-credential-socket\") pod \"28b9fcb2-e588-4d76-909c-244704b9a658\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " Nov 03 15:36:40.828290 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.828276 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b9fcb2-e588-4d76-909c-244704b9a658-kube-api-access-srm69\" (UniqueName: \"kubernetes.io/projected/28b9fcb2-e588-4d76-909c-244704b9a658-kube-api-access-srm69\") pod \"28b9fcb2-e588-4d76-909c-244704b9a658\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " Nov 03 15:36:40.828394 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.828298 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28b9fcb2-e588-4d76-909c-244704b9a658-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b9fcb2-e588-4d76-909c-244704b9a658-istio-podinfo\") pod \"28b9fcb2-e588-4d76-909c-244704b9a658\" (UID: \"28b9fcb2-e588-4d76-909c-244704b9a658\") " Nov 03 15:36:40.828882 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.828856 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-workload-socket" pod "28b9fcb2-e588-4d76-909c-244704b9a658" (UID: "28b9fcb2-e588-4d76-909c-244704b9a658"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:40.829248 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.829227 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.832498 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.832465 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28b9fcb2-e588-4d76-909c-244704b9a658-istio-podinfo" pod "28b9fcb2-e588-4d76-909c-244704b9a658" (UID: "28b9fcb2-e588-4d76-909c-244704b9a658"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:36:40.835110 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.834956 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-istio-envoy" pod "28b9fcb2-e588-4d76-909c-244704b9a658" (UID: "28b9fcb2-e588-4d76-909c-244704b9a658"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:40.835321 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.835301 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b9fcb2-e588-4d76-909c-244704b9a658-istiod-ca-cert" pod "28b9fcb2-e588-4d76-909c-244704b9a658" (UID: "28b9fcb2-e588-4d76-909c-244704b9a658"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:36:40.835525 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.835507 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-istio-data" pod "28b9fcb2-e588-4d76-909c-244704b9a658" (UID: "28b9fcb2-e588-4d76-909c-244704b9a658"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:40.835680 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.835662 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-credential-socket" pod "28b9fcb2-e588-4d76-909c-244704b9a658" (UID: "28b9fcb2-e588-4d76-909c-244704b9a658"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:40.835868 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.835850 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-workload-certs" pod "28b9fcb2-e588-4d76-909c-244704b9a658" (UID: "28b9fcb2-e588-4d76-909c-244704b9a658"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:40.837120 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.837088 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b9fcb2-e588-4d76-909c-244704b9a658-istio-token" pod "28b9fcb2-e588-4d76-909c-244704b9a658" (UID: "28b9fcb2-e588-4d76-909c-244704b9a658"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:36:40.838289 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.838252 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b9fcb2-e588-4d76-909c-244704b9a658-kube-api-access-srm69" pod "28b9fcb2-e588-4d76-909c-244704b9a658" (UID: "28b9fcb2-e588-4d76-909c-244704b9a658"). InnerVolumeSpecName "kube-api-access-srm69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:36:40.932114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.932027 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.932114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.932059 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b9fcb2-e588-4d76-909c-244704b9a658-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.932114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.932072 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.932114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.932085 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.932114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.932098 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srm69\" (UniqueName: \"kubernetes.io/projected/28b9fcb2-e588-4d76-909c-244704b9a658-kube-api-access-srm69\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.932114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.932112 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b9fcb2-e588-4d76-909c-244704b9a658-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.932500 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.932126 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b9fcb2-e588-4d76-909c-244704b9a658-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.932500 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:40.932333 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b9fcb2-e588-4d76-909c-244704b9a658-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.628062 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:41.628035 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" Nov 03 15:36:41.628606 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:41.628066 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4" event={"ID":"28b9fcb2-e588-4d76-909c-244704b9a658","Type":"ContainerDied","Data":"06cbfbe684eb44c6ce2a66ff42b9a954b3cf8627c56203fb47c532a06ee0c9cf"} Nov 03 15:36:41.628606 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:41.628134 4290 scope.go:122] "RemoveContainer" containerID="6f4f017f86a40b69b4bdc4098169dea614ce8fcabd461e3f58db9cb7daa0a122" Nov 03 15:36:41.853492 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:41.853455 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4"] Nov 03 15:36:41.900161 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:41.900084 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-qs2m4"] Nov 03 15:36:42.684258 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:42.684220 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b9fcb2-e588-4d76-909c-244704b9a658" path="/var/lib/kubelet/pods/28b9fcb2-e588-4d76-909c-244704b9a658/volumes" Nov 03 15:36:42.685027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:36:42.685008 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2af12a1b-e8c5-418d-b61d-f9928a8213b6" path="/var/lib/kubelet/pods/2af12a1b-e8c5-418d-b61d-f9928a8213b6/volumes" Nov 03 15:37:08.704755 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:37:08.704634 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:37:16.988786 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:16.988754 4290 scope.go:122] "RemoveContainer" containerID="a26c882371d548259ace46eb2d6163e80a8932c4c4f62f13f27c29b9d8967225" Nov 03 15:37:34.804210 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:34.804174 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk"] Nov 03 15:37:34.804647 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:34.804380 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" podUID="1fda87ab-de4d-4bac-908f-b8c2ad6123e8" containerName="istio-proxy" containerID="cri-o://c2eae0c017aa35ccd1476ae7115cfb7c4b86530e80951ebbdbec64b3c7c50974" gracePeriod=30 Nov 03 15:37:38.716558 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:37:38.716512 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:37:40.120632 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.120600 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:37:40.279570 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.279536 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istiod-ca-cert\") pod \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " Nov 03 15:37:40.279827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.279632 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-podinfo\") pod \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " Nov 03 15:37:40.279827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.279669 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-workload-certs\") pod \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " Nov 03 15:37:40.279965 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.279844 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-workload-socket\") pod \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " Nov 03 15:37:40.280020 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.279991 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-kube-api-access-h2rj9\" (UniqueName: \"kubernetes.io/projected/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-kube-api-access-h2rj9\") pod \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " Nov 03 15:37:40.280076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.280027 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istiod-ca-cert" pod "1fda87ab-de4d-4bac-908f-b8c2ad6123e8" (UID: "1fda87ab-de4d-4bac-908f-b8c2ad6123e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:37:40.280139 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.280050 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-workload-certs" pod "1fda87ab-de4d-4bac-908f-b8c2ad6123e8" (UID: "1fda87ab-de4d-4bac-908f-b8c2ad6123e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:40.280139 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.280128 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-workload-socket" pod "1fda87ab-de4d-4bac-908f-b8c2ad6123e8" (UID: "1fda87ab-de4d-4bac-908f-b8c2ad6123e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:40.280431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.280403 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-data\") pod \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " Nov 03 15:37:40.280579 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.280528 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-token\" (UniqueName: \"kubernetes.io/projected/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-token\") pod \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " Nov 03 15:37:40.280579 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.280563 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-credential-socket\") pod \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " Nov 03 15:37:40.280965 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.280601 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-envoy\") pod \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\" (UID: \"1fda87ab-de4d-4bac-908f-b8c2ad6123e8\") " Nov 03 15:37:40.280965 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.280818 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-data" pod "1fda87ab-de4d-4bac-908f-b8c2ad6123e8" (UID: "1fda87ab-de4d-4bac-908f-b8c2ad6123e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:40.280965 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.280942 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-credential-socket" pod "1fda87ab-de4d-4bac-908f-b8c2ad6123e8" (UID: "1fda87ab-de4d-4bac-908f-b8c2ad6123e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:40.281625 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.281533 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:40.281625 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.281559 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:40.281625 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.281575 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:40.281625 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.281589 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:40.281625 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.281603 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:40.282582 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.282558 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-kube-api-access-h2rj9" pod "1fda87ab-de4d-4bac-908f-b8c2ad6123e8" (UID: "1fda87ab-de4d-4bac-908f-b8c2ad6123e8"). InnerVolumeSpecName "kube-api-access-h2rj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:40.283017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.282952 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-podinfo" pod "1fda87ab-de4d-4bac-908f-b8c2ad6123e8" (UID: "1fda87ab-de4d-4bac-908f-b8c2ad6123e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:37:40.283335 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.283312 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-token" pod "1fda87ab-de4d-4bac-908f-b8c2ad6123e8" (UID: "1fda87ab-de4d-4bac-908f-b8c2ad6123e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:40.283418 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.283367 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-envoy" pod "1fda87ab-de4d-4bac-908f-b8c2ad6123e8" (UID: "1fda87ab-de4d-4bac-908f-b8c2ad6123e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:40.372503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.372468 4290 generic.go:356] "Generic (PLEG): container finished" podID="1fda87ab-de4d-4bac-908f-b8c2ad6123e8" containerID="c2eae0c017aa35ccd1476ae7115cfb7c4b86530e80951ebbdbec64b3c7c50974" exitCode=0 Nov 03 15:37:40.372670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.372523 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" Nov 03 15:37:40.372670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.372562 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" event={"ID":"1fda87ab-de4d-4bac-908f-b8c2ad6123e8","Type":"ContainerDied","Data":"c2eae0c017aa35ccd1476ae7115cfb7c4b86530e80951ebbdbec64b3c7c50974"} Nov 03 15:37:40.372670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.372632 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk" event={"ID":"1fda87ab-de4d-4bac-908f-b8c2ad6123e8","Type":"ContainerDied","Data":"9c5ba570c891f675dc64ed8a36782c8e389dd394bc00bdcb9ee9cd3916993f5d"} Nov 03 15:37:40.372670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.372663 4290 scope.go:122] "RemoveContainer" containerID="c2eae0c017aa35ccd1476ae7115cfb7c4b86530e80951ebbdbec64b3c7c50974" Nov 03 15:37:40.383080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.383053 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:40.383200 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.383085 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2rj9\" (UniqueName: \"kubernetes.io/projected/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-kube-api-access-h2rj9\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:40.383200 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.383103 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:40.383200 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.383132 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fda87ab-de4d-4bac-908f-b8c2ad6123e8-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:40.388976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.388957 4290 scope.go:122] "RemoveContainer" containerID="c2eae0c017aa35ccd1476ae7115cfb7c4b86530e80951ebbdbec64b3c7c50974" Nov 03 15:37:40.389284 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:37:40.389253 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2eae0c017aa35ccd1476ae7115cfb7c4b86530e80951ebbdbec64b3c7c50974\": container with ID starting with c2eae0c017aa35ccd1476ae7115cfb7c4b86530e80951ebbdbec64b3c7c50974 not found: ID does not exist" containerID="c2eae0c017aa35ccd1476ae7115cfb7c4b86530e80951ebbdbec64b3c7c50974" Nov 03 15:37:40.389373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.389296 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2eae0c017aa35ccd1476ae7115cfb7c4b86530e80951ebbdbec64b3c7c50974"} err="failed to get container status \"c2eae0c017aa35ccd1476ae7115cfb7c4b86530e80951ebbdbec64b3c7c50974\": rpc error: code = NotFound desc = could not find container \"c2eae0c017aa35ccd1476ae7115cfb7c4b86530e80951ebbdbec64b3c7c50974\": container with ID starting with c2eae0c017aa35ccd1476ae7115cfb7c4b86530e80951ebbdbec64b3c7c50974 not found: ID does not exist" Nov 03 15:37:40.541495 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.541442 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk"] Nov 03 15:37:40.551174 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.551132 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-wmqgk"] Nov 03 15:37:40.692921 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:37:40.692827 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fda87ab-de4d-4bac-908f-b8c2ad6123e8" path="/var/lib/kubelet/pods/1fda87ab-de4d-4bac-908f-b8c2ad6123e8/volumes" Nov 03 15:38:08.734321 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:38:08.734232 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:38:13.077853 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:13.074730 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:38:13.083044 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:13.083017 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:38:13.095821 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:13.094537 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:38:13.097828 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:13.097795 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:38:23.693462 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:23.693429 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf"] Nov 03 15:38:32.024975 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:32.024940 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x"] Nov 03 15:38:32.161456 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:32.161416 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn"] Nov 03 15:38:37.025632 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:37.025569 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" podUID="c05e39ac-2a6d-4071-887b-ee32f91514a9" containerName="payload-processing" containerID="cri-o://53d77dc8d3e6713526aa91a0ca00333ff616fe8ec485ea2ea8426c72e85d69a1" gracePeriod=25 Nov 03 15:38:37.162240 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:37.162163 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" podUID="0000e843-b56d-4f34-8001-65de8a8faaa8" containerName="payload-pre-processing" containerID="cri-o://97b4781aa30210e5606ba726b42dfbd817e5f17de18f23b1d163fa3bf24a7d38" gracePeriod=25 Nov 03 15:38:38.703879 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:38:38.703840 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:38:38.716065 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:38.716034 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" containerID="cri-o://1c85c6e6264e4d72f5a52d0fe8a1c20cdec5c5d3743146736a3326dc37ed7ca4" gracePeriod=45 Nov 03 15:38:39.147641 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.147458 4290 generic.go:356] "Generic (PLEG): container finished" podID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerID="1c85c6e6264e4d72f5a52d0fe8a1c20cdec5c5d3743146736a3326dc37ed7ca4" exitCode=0 Nov 03 15:38:39.147641 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.147508 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" event={"ID":"ec99faf0-1759-4384-b30f-fb4c2679dfc4","Type":"ContainerDied","Data":"1c85c6e6264e4d72f5a52d0fe8a1c20cdec5c5d3743146736a3326dc37ed7ca4"} Nov 03 15:38:39.163771 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.156214 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:38:39.206310 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.206276 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-dshm\") pod \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " Nov 03 15:38:39.206564 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.206406 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec99faf0-1759-4384-b30f-fb4c2679dfc4-kube-api-access-pqdbk\" (UniqueName: \"kubernetes.io/projected/ec99faf0-1759-4384-b30f-fb4c2679dfc4-kube-api-access-pqdbk\") pod \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " Nov 03 15:38:39.206564 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.206474 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-tmp-dir\") pod \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " Nov 03 15:38:39.206564 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.206533 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-home\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-home\") pod \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " Nov 03 15:38:39.206812 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.206626 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec99faf0-1759-4384-b30f-fb4c2679dfc4-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec99faf0-1759-4384-b30f-fb4c2679dfc4-tls-certs\") pod \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " Nov 03 15:38:39.206812 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.206677 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-model-cache\") pod \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\" (UID: \"ec99faf0-1759-4384-b30f-fb4c2679dfc4\") " Nov 03 15:38:39.206981 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.206951 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-tmp-dir" pod "ec99faf0-1759-4384-b30f-fb4c2679dfc4" (UID: "ec99faf0-1759-4384-b30f-fb4c2679dfc4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:39.207459 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.207424 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-model-cache" pod "ec99faf0-1759-4384-b30f-fb4c2679dfc4" (UID: "ec99faf0-1759-4384-b30f-fb4c2679dfc4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:39.207674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.207649 4290 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-tmp-dir\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.207813 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.207765 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-home" pod "ec99faf0-1759-4384-b30f-fb4c2679dfc4" (UID: "ec99faf0-1759-4384-b30f-fb4c2679dfc4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:39.210340 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.210314 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec99faf0-1759-4384-b30f-fb4c2679dfc4-kube-api-access-pqdbk" pod "ec99faf0-1759-4384-b30f-fb4c2679dfc4" (UID: "ec99faf0-1759-4384-b30f-fb4c2679dfc4"). InnerVolumeSpecName "kube-api-access-pqdbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:39.234832 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.234796 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-dshm" pod "ec99faf0-1759-4384-b30f-fb4c2679dfc4" (UID: "ec99faf0-1759-4384-b30f-fb4c2679dfc4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:39.235245 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.235212 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec99faf0-1759-4384-b30f-fb4c2679dfc4-tls-certs" pod "ec99faf0-1759-4384-b30f-fb4c2679dfc4" (UID: "ec99faf0-1759-4384-b30f-fb4c2679dfc4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:38:39.310118 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.310015 4290 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-dshm\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.310118 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.310059 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqdbk\" (UniqueName: \"kubernetes.io/projected/ec99faf0-1759-4384-b30f-fb4c2679dfc4-kube-api-access-pqdbk\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.310118 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.310076 4290 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-home\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.310118 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.310091 4290 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec99faf0-1759-4384-b30f-fb4c2679dfc4-tls-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.310118 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:39.310105 4290 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec99faf0-1759-4384-b30f-fb4c2679dfc4-model-cache\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:40.158833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:40.158796 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" Nov 03 15:38:40.158833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:40.158828 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf" event={"ID":"ec99faf0-1759-4384-b30f-fb4c2679dfc4","Type":"ContainerDied","Data":"171a10ce935b0865efc78f33583440b213fb8d93950d56b99c2a5507b00ccf77"} Nov 03 15:38:40.159360 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:40.158880 4290 scope.go:122] "RemoveContainer" containerID="1c85c6e6264e4d72f5a52d0fe8a1c20cdec5c5d3743146736a3326dc37ed7ca4" Nov 03 15:38:40.327384 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:40.327343 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf"] Nov 03 15:38:40.332825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:40.332795 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/premium-w2-f78fd6-kserve-58d4cfc99c-rl4qf"] Nov 03 15:38:40.684768 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:40.684692 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" path="/var/lib/kubelet/pods/ec99faf0-1759-4384-b30f-fb4c2679dfc4/volumes" Nov 03 15:38:46.409891 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:46.409845 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" podUID="c05e39ac-2a6d-4071-887b-ee32f91514a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:38:46.410329 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:46.409897 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" podUID="0000e843-b56d-4f34-8001-65de8a8faaa8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:38:50.355557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:50.355516 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2"] Nov 03 15:38:50.356021 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:50.355777 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" podUID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" containerName="istio-proxy" containerID="cri-o://e00ee0890113793576bac8fc281a751b0b26d34cc21911850d4d011c06c1bc7e" gracePeriod=30 Nov 03 15:38:51.741997 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:51.741949 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2"] Nov 03 15:38:51.742485 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:51.742200 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" containerID="cri-o://785ff0e19bcba4146ba263084590b2b2e603fa23bef4dca752506cb4a1e6e396" gracePeriod=30 Nov 03 15:38:55.654745 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.654718 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:38:55.671446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.671419 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-data\") pod \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " Nov 03 15:38:55.671598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.671580 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-kube-api-access-xqt4g\" (UniqueName: \"kubernetes.io/projected/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-kube-api-access-xqt4g\") pod \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " Nov 03 15:38:55.671667 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.671623 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-token\" (UniqueName: \"kubernetes.io/projected/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-token\") pod \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " Nov 03 15:38:55.671778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.671677 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-podinfo\") pod \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " Nov 03 15:38:55.671778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.671730 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-credential-socket\") pod \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " Nov 03 15:38:55.671893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.671806 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-workload-certs\") pod \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " Nov 03 15:38:55.671893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.671840 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-workload-socket\") pod \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " Nov 03 15:38:55.671893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.671694 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-data" pod "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" (UID: "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:55.672044 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.671993 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istiod-ca-cert\") pod \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " Nov 03 15:38:55.672099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.672074 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-envoy\") pod \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\" (UID: \"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b\") " Nov 03 15:38:55.672160 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.672091 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-credential-socket" pod "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" (UID: "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:55.672302 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.672252 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-workload-socket" pod "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" (UID: "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:55.672369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.672324 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-workload-certs" pod "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" (UID: "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:55.672562 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.672533 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istiod-ca-cert" pod "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" (UID: "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:38:55.673178 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.673152 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:55.673275 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.673186 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:55.673275 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.673204 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:55.673275 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.673217 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:55.673275 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.673231 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:55.674140 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.674118 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-token" pod "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" (UID: "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:55.674424 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.674403 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-kube-api-access-xqt4g" pod "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" (UID: "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b"). InnerVolumeSpecName "kube-api-access-xqt4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:55.674946 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.674925 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-envoy" pod "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" (UID: "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:55.675032 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.674961 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-podinfo" pod "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" (UID: "10aaf3eb-a9f5-4930-9fe5-767d65c63d4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:38:55.775021 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.774929 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:55.775021 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.774962 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqt4g\" (UniqueName: \"kubernetes.io/projected/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-kube-api-access-xqt4g\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:55.775021 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.774982 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:55.775021 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:55.774997 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.315593 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:56.315562 4290 generic.go:356] "Generic (PLEG): container finished" podID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" containerID="e00ee0890113793576bac8fc281a751b0b26d34cc21911850d4d011c06c1bc7e" exitCode=0 Nov 03 15:38:56.315793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:56.315624 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" Nov 03 15:38:56.315793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:56.315653 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" event={"ID":"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b","Type":"ContainerDied","Data":"e00ee0890113793576bac8fc281a751b0b26d34cc21911850d4d011c06c1bc7e"} Nov 03 15:38:56.315793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:56.315766 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2" event={"ID":"10aaf3eb-a9f5-4930-9fe5-767d65c63d4b","Type":"ContainerDied","Data":"25a5b8c723b6e9f4782b9b26d1aa239ce95730dd956139969cfa431f13adddd0"} Nov 03 15:38:56.315910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:56.315792 4290 scope.go:122] "RemoveContainer" containerID="e00ee0890113793576bac8fc281a751b0b26d34cc21911850d4d011c06c1bc7e" Nov 03 15:38:56.328772 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:56.328749 4290 scope.go:122] "RemoveContainer" containerID="e00ee0890113793576bac8fc281a751b0b26d34cc21911850d4d011c06c1bc7e" Nov 03 15:38:56.329069 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:38:56.329048 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e00ee0890113793576bac8fc281a751b0b26d34cc21911850d4d011c06c1bc7e\": container with ID starting with e00ee0890113793576bac8fc281a751b0b26d34cc21911850d4d011c06c1bc7e not found: ID does not exist" containerID="e00ee0890113793576bac8fc281a751b0b26d34cc21911850d4d011c06c1bc7e" Nov 03 15:38:56.329127 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:56.329076 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00ee0890113793576bac8fc281a751b0b26d34cc21911850d4d011c06c1bc7e"} err="failed to get container status \"e00ee0890113793576bac8fc281a751b0b26d34cc21911850d4d011c06c1bc7e\": rpc error: code = NotFound desc = could not find container \"e00ee0890113793576bac8fc281a751b0b26d34cc21911850d4d011c06c1bc7e\": container with ID starting with e00ee0890113793576bac8fc281a751b0b26d34cc21911850d4d011c06c1bc7e not found: ID does not exist" Nov 03 15:38:56.409450 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:56.409405 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" podUID="0000e843-b56d-4f34-8001-65de8a8faaa8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:38:56.409450 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:56.409433 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" podUID="c05e39ac-2a6d-4071-887b-ee32f91514a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:38:56.487769 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:56.487728 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2"] Nov 03 15:38:56.538077 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:56.538014 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-rj2v2"] Nov 03 15:38:56.680340 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:56.680300 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" path="/var/lib/kubelet/pods/10aaf3eb-a9f5-4930-9fe5-767d65c63d4b/volumes" Nov 03 15:38:57.134381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.134356 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:38:57.195142 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.195116 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-podinfo\") pod \"6d563b0f-d43c-4799-a59c-cf440af5506f\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " Nov 03 15:38:57.195270 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.195172 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d563b0f-d43c-4799-a59c-cf440af5506f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d563b0f-d43c-4799-a59c-cf440af5506f-istiod-ca-cert\") pod \"6d563b0f-d43c-4799-a59c-cf440af5506f\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " Nov 03 15:38:57.195270 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.195208 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-data\") pod \"6d563b0f-d43c-4799-a59c-cf440af5506f\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " Nov 03 15:38:57.195270 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.195260 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-token\" (UniqueName: \"kubernetes.io/projected/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-token\") pod \"6d563b0f-d43c-4799-a59c-cf440af5506f\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " Nov 03 15:38:57.195537 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.195310 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-envoy\") pod \"6d563b0f-d43c-4799-a59c-cf440af5506f\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " Nov 03 15:38:57.195537 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.195403 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d563b0f-d43c-4799-a59c-cf440af5506f-kube-api-access-zfrll\" (UniqueName: \"kubernetes.io/projected/6d563b0f-d43c-4799-a59c-cf440af5506f-kube-api-access-zfrll\") pod \"6d563b0f-d43c-4799-a59c-cf440af5506f\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " Nov 03 15:38:57.195537 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.195464 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-workload-socket\") pod \"6d563b0f-d43c-4799-a59c-cf440af5506f\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " Nov 03 15:38:57.196124 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.195536 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-data" pod "6d563b0f-d43c-4799-a59c-cf440af5506f" (UID: "6d563b0f-d43c-4799-a59c-cf440af5506f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:57.196124 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.195609 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-workload-certs\") pod \"6d563b0f-d43c-4799-a59c-cf440af5506f\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " Nov 03 15:38:57.196124 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.195651 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-credential-socket\") pod \"6d563b0f-d43c-4799-a59c-cf440af5506f\" (UID: \"6d563b0f-d43c-4799-a59c-cf440af5506f\") " Nov 03 15:38:57.196124 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.195679 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d563b0f-d43c-4799-a59c-cf440af5506f-istiod-ca-cert" pod "6d563b0f-d43c-4799-a59c-cf440af5506f" (UID: "6d563b0f-d43c-4799-a59c-cf440af5506f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:38:57.196124 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.195878 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-workload-socket" pod "6d563b0f-d43c-4799-a59c-cf440af5506f" (UID: "6d563b0f-d43c-4799-a59c-cf440af5506f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:57.196588 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.196287 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-credential-socket" pod "6d563b0f-d43c-4799-a59c-cf440af5506f" (UID: "6d563b0f-d43c-4799-a59c-cf440af5506f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:57.196656 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.196624 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.196656 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.196647 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.196791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.196663 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d563b0f-d43c-4799-a59c-cf440af5506f-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.196791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.196677 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.196791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.196779 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-workload-certs" pod "6d563b0f-d43c-4799-a59c-cf440af5506f" (UID: "6d563b0f-d43c-4799-a59c-cf440af5506f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:57.198242 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.197780 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-podinfo" pod "6d563b0f-d43c-4799-a59c-cf440af5506f" (UID: "6d563b0f-d43c-4799-a59c-cf440af5506f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:38:57.198242 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.198079 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-token" pod "6d563b0f-d43c-4799-a59c-cf440af5506f" (UID: "6d563b0f-d43c-4799-a59c-cf440af5506f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:57.199023 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.198993 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d563b0f-d43c-4799-a59c-cf440af5506f-kube-api-access-zfrll" pod "6d563b0f-d43c-4799-a59c-cf440af5506f" (UID: "6d563b0f-d43c-4799-a59c-cf440af5506f"). InnerVolumeSpecName "kube-api-access-zfrll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:57.199120 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.199051 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-envoy" pod "6d563b0f-d43c-4799-a59c-cf440af5506f" (UID: "6d563b0f-d43c-4799-a59c-cf440af5506f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:57.298019 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.297989 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.298019 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.298022 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.298283 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.298036 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.298283 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.298051 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d563b0f-d43c-4799-a59c-cf440af5506f-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.298283 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.298064 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfrll\" (UniqueName: \"kubernetes.io/projected/6d563b0f-d43c-4799-a59c-cf440af5506f-kube-api-access-zfrll\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.317447 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.317065 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" Nov 03 15:38:57.329837 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.329803 4290 generic.go:356] "Generic (PLEG): container finished" podID="c05e39ac-2a6d-4071-887b-ee32f91514a9" containerID="53d77dc8d3e6713526aa91a0ca00333ff616fe8ec485ea2ea8426c72e85d69a1" exitCode=0 Nov 03 15:38:57.330003 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.329943 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" Nov 03 15:38:57.330068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.330048 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" event={"ID":"c05e39ac-2a6d-4071-887b-ee32f91514a9","Type":"ContainerDied","Data":"53d77dc8d3e6713526aa91a0ca00333ff616fe8ec485ea2ea8426c72e85d69a1"} Nov 03 15:38:57.330130 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.330078 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x" event={"ID":"c05e39ac-2a6d-4071-887b-ee32f91514a9","Type":"ContainerDied","Data":"d589ab679ef5b0d095067c6554f9536aba1e9c8ca3fcd6208510c6013e645a23"} Nov 03 15:38:57.330130 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.330098 4290 scope.go:122] "RemoveContainer" containerID="53d77dc8d3e6713526aa91a0ca00333ff616fe8ec485ea2ea8426c72e85d69a1" Nov 03 15:38:57.334077 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.331594 4290 generic.go:356] "Generic (PLEG): container finished" podID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerID="785ff0e19bcba4146ba263084590b2b2e603fa23bef4dca752506cb4a1e6e396" exitCode=0 Nov 03 15:38:57.334077 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.331754 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" Nov 03 15:38:57.334077 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.332289 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" event={"ID":"6d563b0f-d43c-4799-a59c-cf440af5506f","Type":"ContainerDied","Data":"785ff0e19bcba4146ba263084590b2b2e603fa23bef4dca752506cb4a1e6e396"} Nov 03 15:38:57.334077 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.332331 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2" event={"ID":"6d563b0f-d43c-4799-a59c-cf440af5506f","Type":"ContainerDied","Data":"30ab7a983df15e387efa973c15a4cb24d84dab6627a71aaba24e4eca55b999bf"} Nov 03 15:38:57.340456 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.340432 4290 generic.go:356] "Generic (PLEG): container finished" podID="0000e843-b56d-4f34-8001-65de8a8faaa8" containerID="97b4781aa30210e5606ba726b42dfbd817e5f17de18f23b1d163fa3bf24a7d38" exitCode=0 Nov 03 15:38:57.340533 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.340502 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" event={"ID":"0000e843-b56d-4f34-8001-65de8a8faaa8","Type":"ContainerDied","Data":"97b4781aa30210e5606ba726b42dfbd817e5f17de18f23b1d163fa3bf24a7d38"} Nov 03 15:38:57.342765 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.342732 4290 scope.go:122] "RemoveContainer" containerID="53d77dc8d3e6713526aa91a0ca00333ff616fe8ec485ea2ea8426c72e85d69a1" Nov 03 15:38:57.343028 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:38:57.343010 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d77dc8d3e6713526aa91a0ca00333ff616fe8ec485ea2ea8426c72e85d69a1\": container with ID starting with 53d77dc8d3e6713526aa91a0ca00333ff616fe8ec485ea2ea8426c72e85d69a1 not found: ID does not exist" containerID="53d77dc8d3e6713526aa91a0ca00333ff616fe8ec485ea2ea8426c72e85d69a1" Nov 03 15:38:57.343085 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.343037 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d77dc8d3e6713526aa91a0ca00333ff616fe8ec485ea2ea8426c72e85d69a1"} err="failed to get container status \"53d77dc8d3e6713526aa91a0ca00333ff616fe8ec485ea2ea8426c72e85d69a1\": rpc error: code = NotFound desc = could not find container \"53d77dc8d3e6713526aa91a0ca00333ff616fe8ec485ea2ea8426c72e85d69a1\": container with ID starting with 53d77dc8d3e6713526aa91a0ca00333ff616fe8ec485ea2ea8426c72e85d69a1 not found: ID does not exist" Nov 03 15:38:57.343085 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.343054 4290 scope.go:122] "RemoveContainer" containerID="785ff0e19bcba4146ba263084590b2b2e603fa23bef4dca752506cb4a1e6e396" Nov 03 15:38:57.354682 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.354662 4290 scope.go:122] "RemoveContainer" containerID="785ff0e19bcba4146ba263084590b2b2e603fa23bef4dca752506cb4a1e6e396" Nov 03 15:38:57.355001 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:38:57.354980 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"785ff0e19bcba4146ba263084590b2b2e603fa23bef4dca752506cb4a1e6e396\": container with ID starting with 785ff0e19bcba4146ba263084590b2b2e603fa23bef4dca752506cb4a1e6e396 not found: ID does not exist" containerID="785ff0e19bcba4146ba263084590b2b2e603fa23bef4dca752506cb4a1e6e396" Nov 03 15:38:57.355080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.355010 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785ff0e19bcba4146ba263084590b2b2e603fa23bef4dca752506cb4a1e6e396"} err="failed to get container status \"785ff0e19bcba4146ba263084590b2b2e603fa23bef4dca752506cb4a1e6e396\": rpc error: code = NotFound desc = could not find container \"785ff0e19bcba4146ba263084590b2b2e603fa23bef4dca752506cb4a1e6e396\": container with ID starting with 785ff0e19bcba4146ba263084590b2b2e603fa23bef4dca752506cb4a1e6e396 not found: ID does not exist" Nov 03 15:38:57.399174 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.399136 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05e39ac-2a6d-4071-887b-ee32f91514a9-kube-api-access-zzrld\" (UniqueName: \"kubernetes.io/projected/c05e39ac-2a6d-4071-887b-ee32f91514a9-kube-api-access-zzrld\") pod \"c05e39ac-2a6d-4071-887b-ee32f91514a9\" (UID: \"c05e39ac-2a6d-4071-887b-ee32f91514a9\") " Nov 03 15:38:57.399410 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.399393 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c05e39ac-2a6d-4071-887b-ee32f91514a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05e39ac-2a6d-4071-887b-ee32f91514a9-plugins-config-volume\") pod \"c05e39ac-2a6d-4071-887b-ee32f91514a9\" (UID: \"c05e39ac-2a6d-4071-887b-ee32f91514a9\") " Nov 03 15:38:57.399862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.399834 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05e39ac-2a6d-4071-887b-ee32f91514a9-plugins-config-volume" pod "c05e39ac-2a6d-4071-887b-ee32f91514a9" (UID: "c05e39ac-2a6d-4071-887b-ee32f91514a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:38:57.400140 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.400124 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05e39ac-2a6d-4071-887b-ee32f91514a9-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.401465 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.401441 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05e39ac-2a6d-4071-887b-ee32f91514a9-kube-api-access-zzrld" pod "c05e39ac-2a6d-4071-887b-ee32f91514a9" (UID: "c05e39ac-2a6d-4071-887b-ee32f91514a9"). InnerVolumeSpecName "kube-api-access-zzrld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:57.449277 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.449242 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" Nov 03 15:38:57.501318 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.501290 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0000e843-b56d-4f34-8001-65de8a8faaa8-kube-api-access-n8js5\" (UniqueName: \"kubernetes.io/projected/0000e843-b56d-4f34-8001-65de8a8faaa8-kube-api-access-n8js5\") pod \"0000e843-b56d-4f34-8001-65de8a8faaa8\" (UID: \"0000e843-b56d-4f34-8001-65de8a8faaa8\") " Nov 03 15:38:57.501622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.501593 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0000e843-b56d-4f34-8001-65de8a8faaa8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0000e843-b56d-4f34-8001-65de8a8faaa8-plugins-config-volume\") pod \"0000e843-b56d-4f34-8001-65de8a8faaa8\" (UID: \"0000e843-b56d-4f34-8001-65de8a8faaa8\") " Nov 03 15:38:57.501942 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.501919 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0000e843-b56d-4f34-8001-65de8a8faaa8-plugins-config-volume" pod "0000e843-b56d-4f34-8001-65de8a8faaa8" (UID: "0000e843-b56d-4f34-8001-65de8a8faaa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:38:57.502147 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.502132 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzrld\" (UniqueName: \"kubernetes.io/projected/c05e39ac-2a6d-4071-887b-ee32f91514a9-kube-api-access-zzrld\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.502215 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.502150 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0000e843-b56d-4f34-8001-65de8a8faaa8-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.503567 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.503546 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0000e843-b56d-4f34-8001-65de8a8faaa8-kube-api-access-n8js5" pod "0000e843-b56d-4f34-8001-65de8a8faaa8" (UID: "0000e843-b56d-4f34-8001-65de8a8faaa8"). InnerVolumeSpecName "kube-api-access-n8js5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:57.603602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.603527 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8js5\" (UniqueName: \"kubernetes.io/projected/0000e843-b56d-4f34-8001-65de8a8faaa8-kube-api-access-n8js5\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.611948 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.611912 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2"] Nov 03 15:38:57.616345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.616320 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-64dx2"] Nov 03 15:38:57.853973 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.853855 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x"] Nov 03 15:38:57.861960 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:57.861927 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f78fd6-6ddb7dc8b5-lsx4x"] Nov 03 15:38:58.373076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:58.373043 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" Nov 03 15:38:58.373223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:58.373046 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn" event={"ID":"0000e843-b56d-4f34-8001-65de8a8faaa8","Type":"ContainerDied","Data":"edbb559441449df79f6a126e86e94717bffe18c22910a33d453e03583c6d04ee"} Nov 03 15:38:58.373297 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:58.373235 4290 scope.go:122] "RemoveContainer" containerID="97b4781aa30210e5606ba726b42dfbd817e5f17de18f23b1d163fa3bf24a7d38" Nov 03 15:38:58.542133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:58.542100 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn"] Nov 03 15:38:58.591516 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:58.591474 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f78fd6-7659c5b847-tlxgn"] Nov 03 15:38:58.681468 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:58.681398 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0000e843-b56d-4f34-8001-65de8a8faaa8" path="/var/lib/kubelet/pods/0000e843-b56d-4f34-8001-65de8a8faaa8/volumes" Nov 03 15:38:58.681918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:58.681902 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" path="/var/lib/kubelet/pods/6d563b0f-d43c-4799-a59c-cf440af5506f/volumes" Nov 03 15:38:58.682311 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:38:58.682287 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c05e39ac-2a6d-4071-887b-ee32f91514a9" path="/var/lib/kubelet/pods/c05e39ac-2a6d-4071-887b-ee32f91514a9/volumes" Nov 03 15:39:08.707355 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:39:08.707270 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:39:09.358616 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.358583 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn"] Nov 03 15:39:09.358864 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.358838 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" podUID="9e53c44e-f134-4562-990b-19a120e0bef0" containerName="discovery" containerID="cri-o://f1015a57ceb279fe0efbd3090b531b221697287ef5cbfe047bf286121a123035" gracePeriod=30 Nov 03 15:39:09.516252 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.516207 4290 generic.go:356] "Generic (PLEG): container finished" podID="9e53c44e-f134-4562-990b-19a120e0bef0" containerID="f1015a57ceb279fe0efbd3090b531b221697287ef5cbfe047bf286121a123035" exitCode=0 Nov 03 15:39:09.516414 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.516284 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" event={"ID":"9e53c44e-f134-4562-990b-19a120e0bef0","Type":"ContainerDied","Data":"f1015a57ceb279fe0efbd3090b531b221697287ef5cbfe047bf286121a123035"} Nov 03 15:39:09.689905 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.689877 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:39:09.803153 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.803124 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-istio-kubeconfig\") pod \"9e53c44e-f134-4562-990b-19a120e0bef0\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " Nov 03 15:39:09.803589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.803171 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e53c44e-f134-4562-990b-19a120e0bef0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e53c44e-f134-4562-990b-19a120e0bef0-istio-csr-ca-configmap\") pod \"9e53c44e-f134-4562-990b-19a120e0bef0\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " Nov 03 15:39:09.803589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.803251 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e53c44e-f134-4562-990b-19a120e0bef0-kube-api-access-ckchk\" (UniqueName: \"kubernetes.io/projected/9e53c44e-f134-4562-990b-19a120e0bef0-kube-api-access-ckchk\") pod \"9e53c44e-f134-4562-990b-19a120e0bef0\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " Nov 03 15:39:09.803589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.803316 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e53c44e-f134-4562-990b-19a120e0bef0-istio-token\" (UniqueName: \"kubernetes.io/projected/9e53c44e-f134-4562-990b-19a120e0bef0-istio-token\") pod \"9e53c44e-f134-4562-990b-19a120e0bef0\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " Nov 03 15:39:09.803589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.803363 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-istio-csr-dns-cert\") pod \"9e53c44e-f134-4562-990b-19a120e0bef0\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " Nov 03 15:39:09.803589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.803415 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e53c44e-f134-4562-990b-19a120e0bef0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e53c44e-f134-4562-990b-19a120e0bef0-local-certs\") pod \"9e53c44e-f134-4562-990b-19a120e0bef0\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " Nov 03 15:39:09.803944 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.803580 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-cacerts\" (UniqueName: \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-cacerts\") pod \"9e53c44e-f134-4562-990b-19a120e0bef0\" (UID: \"9e53c44e-f134-4562-990b-19a120e0bef0\") " Nov 03 15:39:09.803944 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.803591 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e53c44e-f134-4562-990b-19a120e0bef0-istio-csr-ca-configmap" pod "9e53c44e-f134-4562-990b-19a120e0bef0" (UID: "9e53c44e-f134-4562-990b-19a120e0bef0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:39:09.804846 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.804794 4290 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e53c44e-f134-4562-990b-19a120e0bef0-istio-csr-ca-configmap\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:09.806149 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.806071 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e53c44e-f134-4562-990b-19a120e0bef0-kube-api-access-ckchk" pod "9e53c44e-f134-4562-990b-19a120e0bef0" (UID: "9e53c44e-f134-4562-990b-19a120e0bef0"). InnerVolumeSpecName "kube-api-access-ckchk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:39:09.806249 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.806170 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-istio-kubeconfig" pod "9e53c44e-f134-4562-990b-19a120e0bef0" (UID: "9e53c44e-f134-4562-990b-19a120e0bef0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:39:09.806383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.806354 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-istio-csr-dns-cert" pod "9e53c44e-f134-4562-990b-19a120e0bef0" (UID: "9e53c44e-f134-4562-990b-19a120e0bef0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:39:09.806483 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.806399 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e53c44e-f134-4562-990b-19a120e0bef0-local-certs" pod "9e53c44e-f134-4562-990b-19a120e0bef0" (UID: "9e53c44e-f134-4562-990b-19a120e0bef0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:39:09.806483 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.806402 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-cacerts" pod "9e53c44e-f134-4562-990b-19a120e0bef0" (UID: "9e53c44e-f134-4562-990b-19a120e0bef0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:39:09.806556 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.806495 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e53c44e-f134-4562-990b-19a120e0bef0-istio-token" pod "9e53c44e-f134-4562-990b-19a120e0bef0" (UID: "9e53c44e-f134-4562-990b-19a120e0bef0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:39:09.905951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.905878 4290 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-cacerts\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:09.905951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.905902 4290 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-istio-kubeconfig\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:09.905951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.905912 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckchk\" (UniqueName: \"kubernetes.io/projected/9e53c44e-f134-4562-990b-19a120e0bef0-kube-api-access-ckchk\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:09.905951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.905923 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e53c44e-f134-4562-990b-19a120e0bef0-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:09.905951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.905932 4290 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e53c44e-f134-4562-990b-19a120e0bef0-istio-csr-dns-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:09.905951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:09.905940 4290 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e53c44e-f134-4562-990b-19a120e0bef0-local-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:10.526328 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:10.526296 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" Nov 03 15:39:10.526633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:10.526332 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn" event={"ID":"9e53c44e-f134-4562-990b-19a120e0bef0","Type":"ContainerDied","Data":"7bd49c08cb6cfda1621bf17987a5aa7b8304e7fe8582d848494fea99c45d8a1d"} Nov 03 15:39:10.526633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:10.526390 4290 scope.go:122] "RemoveContainer" containerID="f1015a57ceb279fe0efbd3090b531b221697287ef5cbfe047bf286121a123035" Nov 03 15:39:10.751811 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:10.751777 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn"] Nov 03 15:39:10.799551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:10.799471 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xk5cn"] Nov 03 15:39:12.681376 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:12.681338 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e53c44e-f134-4562-990b-19a120e0bef0" path="/var/lib/kubelet/pods/9e53c44e-f134-4562-990b-19a120e0bef0/volumes" Nov 03 15:39:15.640080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640049 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv"] Nov 03 15:39:15.640829 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640810 4290 cpu_manager.go:414] "containerMap: removing container" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" Nov 03 15:39:15.640881 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640830 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" Nov 03 15:39:15.640881 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640851 4290 cpu_manager.go:414] "containerMap: removing container" podUID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" containerName="istio-proxy" Nov 03 15:39:15.640881 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640856 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" containerName="istio-proxy" Nov 03 15:39:15.640881 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640866 4290 cpu_manager.go:414] "containerMap: removing container" podUID="9e53c44e-f134-4562-990b-19a120e0bef0" containerName="discovery" Nov 03 15:39:15.640881 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640871 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e53c44e-f134-4562-990b-19a120e0bef0" containerName="discovery" Nov 03 15:39:15.640881 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640881 4290 cpu_manager.go:414] "containerMap: removing container" podUID="1fda87ab-de4d-4bac-908f-b8c2ad6123e8" containerName="istio-proxy" Nov 03 15:39:15.641075 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640886 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fda87ab-de4d-4bac-908f-b8c2ad6123e8" containerName="istio-proxy" Nov 03 15:39:15.641075 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640901 4290 cpu_manager.go:414] "containerMap: removing container" podUID="0000e843-b56d-4f34-8001-65de8a8faaa8" containerName="payload-pre-processing" Nov 03 15:39:15.641075 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640906 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0000e843-b56d-4f34-8001-65de8a8faaa8" containerName="payload-pre-processing" Nov 03 15:39:15.641075 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640919 4290 cpu_manager.go:414] "containerMap: removing container" podUID="2af12a1b-e8c5-418d-b61d-f9928a8213b6" containerName="istio-proxy" Nov 03 15:39:15.641075 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640926 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af12a1b-e8c5-418d-b61d-f9928a8213b6" containerName="istio-proxy" Nov 03 15:39:15.641075 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640936 4290 cpu_manager.go:414] "containerMap: removing container" podUID="28b9fcb2-e588-4d76-909c-244704b9a658" containerName="istio-proxy" Nov 03 15:39:15.641075 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640941 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b9fcb2-e588-4d76-909c-244704b9a658" containerName="istio-proxy" Nov 03 15:39:15.641075 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640947 4290 cpu_manager.go:414] "containerMap: removing container" podUID="c05e39ac-2a6d-4071-887b-ee32f91514a9" containerName="payload-processing" Nov 03 15:39:15.641075 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640951 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05e39ac-2a6d-4071-887b-ee32f91514a9" containerName="payload-processing" Nov 03 15:39:15.641075 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640959 4290 cpu_manager.go:414] "containerMap: removing container" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" Nov 03 15:39:15.641075 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.640963 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" Nov 03 15:39:15.641075 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.641067 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="10aaf3eb-a9f5-4930-9fe5-767d65c63d4b" containerName="istio-proxy" Nov 03 15:39:15.641075 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.641076 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e53c44e-f134-4562-990b-19a120e0bef0" containerName="discovery" Nov 03 15:39:15.641446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.641085 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fda87ab-de4d-4bac-908f-b8c2ad6123e8" containerName="istio-proxy" Nov 03 15:39:15.641446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.641096 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="0000e843-b56d-4f34-8001-65de8a8faaa8" containerName="payload-pre-processing" Nov 03 15:39:15.641446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.641106 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="2af12a1b-e8c5-418d-b61d-f9928a8213b6" containerName="istio-proxy" Nov 03 15:39:15.641446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.641113 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b9fcb2-e588-4d76-909c-244704b9a658" containerName="istio-proxy" Nov 03 15:39:15.641446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.641119 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05e39ac-2a6d-4071-887b-ee32f91514a9" containerName="payload-processing" Nov 03 15:39:15.641446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.641125 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d563b0f-d43c-4799-a59c-cf440af5506f" containerName="istio-proxy" Nov 03 15:39:15.641446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.641134 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec99faf0-1759-4384-b30f-fb4c2679dfc4" containerName="main" Nov 03 15:39:15.648774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.648260 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" Nov 03 15:39:15.706081 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.706051 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv"] Nov 03 15:39:15.731853 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.731815 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj74l\" (UniqueName: \"kubernetes.io/projected/7a52bd2f-9d0b-4782-9e30-da3861709062-kube-api-access-qj74l\") pod \"maas-api-revoke-sub-5c8eea00-ptmpv\" (UID: \"7a52bd2f-9d0b-4782-9e30-da3861709062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" Nov 03 15:39:15.732031 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.731924 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a52bd2f-9d0b-4782-9e30-da3861709062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c8eea00-ptmpv\" (UID: \"7a52bd2f-9d0b-4782-9e30-da3861709062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" Nov 03 15:39:15.732114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.732097 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a52bd2f-9d0b-4782-9e30-da3861709062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c8eea00-ptmpv\" (UID: \"7a52bd2f-9d0b-4782-9e30-da3861709062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" Nov 03 15:39:15.751186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.751153 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:39:15.751379 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.751284 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:39:15.833853 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.833823 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj74l\" (UniqueName: \"kubernetes.io/projected/7a52bd2f-9d0b-4782-9e30-da3861709062-kube-api-access-qj74l\") pod \"maas-api-revoke-sub-5c8eea00-ptmpv\" (UID: \"7a52bd2f-9d0b-4782-9e30-da3861709062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" Nov 03 15:39:15.834045 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.833883 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a52bd2f-9d0b-4782-9e30-da3861709062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c8eea00-ptmpv\" (UID: \"7a52bd2f-9d0b-4782-9e30-da3861709062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" Nov 03 15:39:15.834045 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.834028 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a52bd2f-9d0b-4782-9e30-da3861709062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c8eea00-ptmpv\" (UID: \"7a52bd2f-9d0b-4782-9e30-da3861709062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" Nov 03 15:39:15.834651 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.834620 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a52bd2f-9d0b-4782-9e30-da3861709062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c8eea00-ptmpv\" (UID: \"7a52bd2f-9d0b-4782-9e30-da3861709062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" Nov 03 15:39:15.836448 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.836421 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a52bd2f-9d0b-4782-9e30-da3861709062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c8eea00-ptmpv\" (UID: \"7a52bd2f-9d0b-4782-9e30-da3861709062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" Nov 03 15:39:15.896221 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.896149 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj74l\" (UniqueName: \"kubernetes.io/projected/7a52bd2f-9d0b-4782-9e30-da3861709062-kube-api-access-qj74l\") pod \"maas-api-revoke-sub-5c8eea00-ptmpv\" (UID: \"7a52bd2f-9d0b-4782-9e30-da3861709062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" Nov 03 15:39:15.974475 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:15.974436 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" Nov 03 15:39:16.366736 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:16.366679 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv"] Nov 03 15:39:16.371348 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:39:16.367867 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a52bd2f_9d0b_4782_9e30_da3861709062.slice/crio-02f77a6fa8c6f95f3a55cd5f5515808a5711b7e62c75ccbaed86ab9939bf7c7c WatchSource:0}: Error finding container 02f77a6fa8c6f95f3a55cd5f5515808a5711b7e62c75ccbaed86ab9939bf7c7c: Status 404 returned error can't find the container with id 02f77a6fa8c6f95f3a55cd5f5515808a5711b7e62c75ccbaed86ab9939bf7c7c Nov 03 15:39:16.595712 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:16.595665 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" event={"ID":"7a52bd2f-9d0b-4782-9e30-da3861709062","Type":"ContainerStarted","Data":"02f77a6fa8c6f95f3a55cd5f5515808a5711b7e62c75ccbaed86ab9939bf7c7c"} Nov 03 15:39:17.647956 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:17.647913 4290 generic.go:356] "Generic (PLEG): container finished" podID="7a52bd2f-9d0b-4782-9e30-da3861709062" containerID="065b9f5d2184d16541b053bc37473474da678848b312d9d67cf000a2d94577c3" exitCode=0 Nov 03 15:39:17.664738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:17.656303 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" event={"ID":"7a52bd2f-9d0b-4782-9e30-da3861709062","Type":"ContainerDied","Data":"065b9f5d2184d16541b053bc37473474da678848b312d9d67cf000a2d94577c3"} Nov 03 15:39:18.965919 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:18.965894 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" Nov 03 15:39:19.024787 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:19.024742 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a52bd2f-9d0b-4782-9e30-da3861709062-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a52bd2f-9d0b-4782-9e30-da3861709062-maas-api-openshift-ca\") pod \"7a52bd2f-9d0b-4782-9e30-da3861709062\" (UID: \"7a52bd2f-9d0b-4782-9e30-da3861709062\") " Nov 03 15:39:19.024983 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:19.024903 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a52bd2f-9d0b-4782-9e30-da3861709062-kube-api-access-qj74l\" (UniqueName: \"kubernetes.io/projected/7a52bd2f-9d0b-4782-9e30-da3861709062-kube-api-access-qj74l\") pod \"7a52bd2f-9d0b-4782-9e30-da3861709062\" (UID: \"7a52bd2f-9d0b-4782-9e30-da3861709062\") " Nov 03 15:39:19.025051 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:19.024989 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a52bd2f-9d0b-4782-9e30-da3861709062-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a52bd2f-9d0b-4782-9e30-da3861709062-maas-api-xks-ca\") pod \"7a52bd2f-9d0b-4782-9e30-da3861709062\" (UID: \"7a52bd2f-9d0b-4782-9e30-da3861709062\") " Nov 03 15:39:19.025157 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:19.025127 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a52bd2f-9d0b-4782-9e30-da3861709062-maas-api-openshift-ca" pod "7a52bd2f-9d0b-4782-9e30-da3861709062" (UID: "7a52bd2f-9d0b-4782-9e30-da3861709062"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:39:19.025611 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:19.025586 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a52bd2f-9d0b-4782-9e30-da3861709062-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:19.027680 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:19.027650 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a52bd2f-9d0b-4782-9e30-da3861709062-maas-api-xks-ca" pod "7a52bd2f-9d0b-4782-9e30-da3861709062" (UID: "7a52bd2f-9d0b-4782-9e30-da3861709062"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:39:19.028234 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:19.028207 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a52bd2f-9d0b-4782-9e30-da3861709062-kube-api-access-qj74l" pod "7a52bd2f-9d0b-4782-9e30-da3861709062" (UID: "7a52bd2f-9d0b-4782-9e30-da3861709062"). InnerVolumeSpecName "kube-api-access-qj74l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:39:19.127719 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:19.127665 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a52bd2f-9d0b-4782-9e30-da3861709062-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:19.127719 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:19.127711 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj74l\" (UniqueName: \"kubernetes.io/projected/7a52bd2f-9d0b-4782-9e30-da3861709062-kube-api-access-qj74l\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:19.696404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:19.696344 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" event={"ID":"7a52bd2f-9d0b-4782-9e30-da3861709062","Type":"ContainerDied","Data":"02f77a6fa8c6f95f3a55cd5f5515808a5711b7e62c75ccbaed86ab9939bf7c7c"} Nov 03 15:39:19.696812 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:19.696767 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02f77a6fa8c6f95f3a55cd5f5515808a5711b7e62c75ccbaed86ab9939bf7c7c" Nov 03 15:39:19.702155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:19.697148 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv" Nov 03 15:39:38.703406 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:39:38.703370 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:39:56.666368 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.666328 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p"] Nov 03 15:39:56.667153 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.667132 4290 cpu_manager.go:414] "containerMap: removing container" podUID="7a52bd2f-9d0b-4782-9e30-da3861709062" containerName="revoke-keys" Nov 03 15:39:56.667153 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.667152 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a52bd2f-9d0b-4782-9e30-da3861709062" containerName="revoke-keys" Nov 03 15:39:56.667314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.667252 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a52bd2f-9d0b-4782-9e30-da3861709062" containerName="revoke-keys" Nov 03 15:39:56.671736 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.671673 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" Nov 03 15:39:56.730370 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.730338 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p"] Nov 03 15:39:56.735188 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.735151 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928cb260-2bf9-4c3e-b72d-cf22412df2ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p\" (UID: \"928cb260-2bf9-4c3e-b72d-cf22412df2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" Nov 03 15:39:56.735438 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.735400 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928cb260-2bf9-4c3e-b72d-cf22412df2ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p\" (UID: \"928cb260-2bf9-4c3e-b72d-cf22412df2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" Nov 03 15:39:56.735539 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.735510 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqmqw\" (UniqueName: \"kubernetes.io/projected/928cb260-2bf9-4c3e-b72d-cf22412df2ff-kube-api-access-bqmqw\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p\" (UID: \"928cb260-2bf9-4c3e-b72d-cf22412df2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" Nov 03 15:39:56.772442 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.772406 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:39:56.772639 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.772469 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:39:56.836607 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.836566 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928cb260-2bf9-4c3e-b72d-cf22412df2ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p\" (UID: \"928cb260-2bf9-4c3e-b72d-cf22412df2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" Nov 03 15:39:56.836827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.836757 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928cb260-2bf9-4c3e-b72d-cf22412df2ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p\" (UID: \"928cb260-2bf9-4c3e-b72d-cf22412df2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" Nov 03 15:39:56.836900 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.836849 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqmqw\" (UniqueName: \"kubernetes.io/projected/928cb260-2bf9-4c3e-b72d-cf22412df2ff-kube-api-access-bqmqw\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p\" (UID: \"928cb260-2bf9-4c3e-b72d-cf22412df2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" Nov 03 15:39:56.837833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.837805 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928cb260-2bf9-4c3e-b72d-cf22412df2ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p\" (UID: \"928cb260-2bf9-4c3e-b72d-cf22412df2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" Nov 03 15:39:56.840341 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.840314 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928cb260-2bf9-4c3e-b72d-cf22412df2ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p\" (UID: \"928cb260-2bf9-4c3e-b72d-cf22412df2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" Nov 03 15:39:56.899106 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.899065 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqmqw\" (UniqueName: \"kubernetes.io/projected/928cb260-2bf9-4c3e-b72d-cf22412df2ff-kube-api-access-bqmqw\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p\" (UID: \"928cb260-2bf9-4c3e-b72d-cf22412df2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" Nov 03 15:39:56.991619 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:56.991310 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" Nov 03 15:39:57.594731 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:57.566634 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p"] Nov 03 15:39:57.698787 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:57.693754 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r"] Nov 03 15:39:57.708259 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:57.708230 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r"] Nov 03 15:39:57.708619 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:57.708548 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" Nov 03 15:39:57.764264 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:57.764139 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-78034f-dockercfg-nmv8s\"" Nov 03 15:39:57.764264 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:57.764232 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-78034f\"" Nov 03 15:39:57.834787 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:57.834268 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0041b317-bb1c-4a54-ae90-ac6078087205-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r\" (UID: \"0041b317-bb1c-4a54-ae90-ac6078087205\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" Nov 03 15:39:57.834787 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:57.834375 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvdn2\" (UniqueName: \"kubernetes.io/projected/0041b317-bb1c-4a54-ae90-ac6078087205-kube-api-access-fvdn2\") pod \"payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r\" (UID: \"0041b317-bb1c-4a54-ae90-ac6078087205\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" Nov 03 15:39:57.957724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:57.956803 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0041b317-bb1c-4a54-ae90-ac6078087205-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r\" (UID: \"0041b317-bb1c-4a54-ae90-ac6078087205\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" Nov 03 15:39:57.957724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:57.956904 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvdn2\" (UniqueName: \"kubernetes.io/projected/0041b317-bb1c-4a54-ae90-ac6078087205-kube-api-access-fvdn2\") pod \"payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r\" (UID: \"0041b317-bb1c-4a54-ae90-ac6078087205\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" Nov 03 15:39:57.966728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:57.961084 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0041b317-bb1c-4a54-ae90-ac6078087205-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r\" (UID: \"0041b317-bb1c-4a54-ae90-ac6078087205\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" Nov 03 15:39:58.061775 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:58.059530 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvdn2\" (UniqueName: \"kubernetes.io/projected/0041b317-bb1c-4a54-ae90-ac6078087205-kube-api-access-fvdn2\") pod \"payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r\" (UID: \"0041b317-bb1c-4a54-ae90-ac6078087205\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" Nov 03 15:39:58.102761 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:58.102526 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" event={"ID":"928cb260-2bf9-4c3e-b72d-cf22412df2ff","Type":"ContainerStarted","Data":"e976964ffec3e58c7ba9bf9e7bffc685015d1ebf2dbf3672de7e49e80507787d"} Nov 03 15:39:58.134061 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:58.134030 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" Nov 03 15:39:58.531138 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:39:58.531101 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0041b317_bb1c_4a54_ae90_ac6078087205.slice/crio-1bf381844f1a121bcfc7c075637e784ea6545698133efca661d13d80d02dce27 WatchSource:0}: Error finding container 1bf381844f1a121bcfc7c075637e784ea6545698133efca661d13d80d02dce27: Status 404 returned error can't find the container with id 1bf381844f1a121bcfc7c075637e784ea6545698133efca661d13d80d02dce27 Nov 03 15:39:58.533420 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:58.533394 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r"] Nov 03 15:39:59.112949 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:59.112864 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" event={"ID":"0041b317-bb1c-4a54-ae90-ac6078087205","Type":"ContainerStarted","Data":"681778167ad5fdc1d480ed3feeddc655672567fe1dbc7617c598bd293cdea5f2"} Nov 03 15:39:59.112949 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:59.112928 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" Nov 03 15:39:59.112949 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:59.112947 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" event={"ID":"0041b317-bb1c-4a54-ae90-ac6078087205","Type":"ContainerStarted","Data":"1bf381844f1a121bcfc7c075637e784ea6545698133efca661d13d80d02dce27"} Nov 03 15:39:59.114776 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:59.114746 4290 generic.go:356] "Generic (PLEG): container finished" podID="928cb260-2bf9-4c3e-b72d-cf22412df2ff" containerID="c03b42a2a365210985cbdd4fb254c6c91119e73a9ca8fe9a03ef4c47817e84b4" exitCode=0 Nov 03 15:39:59.114915 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:59.114814 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" event={"ID":"928cb260-2bf9-4c3e-b72d-cf22412df2ff","Type":"ContainerDied","Data":"c03b42a2a365210985cbdd4fb254c6c91119e73a9ca8fe9a03ef4c47817e84b4"} Nov 03 15:39:59.237372 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:39:59.237315 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" podStartSLOduration=2.23730115 podStartE2EDuration="2.23730115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:39:59.235047803 +0000 UTC m=+1906.666005610" watchObservedRunningTime="2026-11-03 15:39:59.23730115 +0000 UTC m=+1906.668258956" Nov 03 15:40:00.300890 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:00.300864 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" Nov 03 15:40:00.414870 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:00.414786 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/928cb260-2bf9-4c3e-b72d-cf22412df2ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928cb260-2bf9-4c3e-b72d-cf22412df2ff-maas-api-xks-ca\") pod \"928cb260-2bf9-4c3e-b72d-cf22412df2ff\" (UID: \"928cb260-2bf9-4c3e-b72d-cf22412df2ff\") " Nov 03 15:40:00.414870 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:00.414855 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928cb260-2bf9-4c3e-b72d-cf22412df2ff-kube-api-access-bqmqw\" (UniqueName: \"kubernetes.io/projected/928cb260-2bf9-4c3e-b72d-cf22412df2ff-kube-api-access-bqmqw\") pod \"928cb260-2bf9-4c3e-b72d-cf22412df2ff\" (UID: \"928cb260-2bf9-4c3e-b72d-cf22412df2ff\") " Nov 03 15:40:00.415112 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:00.414951 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/928cb260-2bf9-4c3e-b72d-cf22412df2ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928cb260-2bf9-4c3e-b72d-cf22412df2ff-maas-api-openshift-ca\") pod \"928cb260-2bf9-4c3e-b72d-cf22412df2ff\" (UID: \"928cb260-2bf9-4c3e-b72d-cf22412df2ff\") " Nov 03 15:40:00.415375 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:00.415343 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/928cb260-2bf9-4c3e-b72d-cf22412df2ff-maas-api-openshift-ca" pod "928cb260-2bf9-4c3e-b72d-cf22412df2ff" (UID: "928cb260-2bf9-4c3e-b72d-cf22412df2ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:40:00.415738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:00.415693 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928cb260-2bf9-4c3e-b72d-cf22412df2ff-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:00.417135 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:00.417110 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/928cb260-2bf9-4c3e-b72d-cf22412df2ff-maas-api-xks-ca" pod "928cb260-2bf9-4c3e-b72d-cf22412df2ff" (UID: "928cb260-2bf9-4c3e-b72d-cf22412df2ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:40:00.417242 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:00.417141 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928cb260-2bf9-4c3e-b72d-cf22412df2ff-kube-api-access-bqmqw" pod "928cb260-2bf9-4c3e-b72d-cf22412df2ff" (UID: "928cb260-2bf9-4c3e-b72d-cf22412df2ff"). InnerVolumeSpecName "kube-api-access-bqmqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:40:00.516751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:00.516719 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928cb260-2bf9-4c3e-b72d-cf22412df2ff-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:00.516751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:00.516750 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqmqw\" (UniqueName: \"kubernetes.io/projected/928cb260-2bf9-4c3e-b72d-cf22412df2ff-kube-api-access-bqmqw\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:01.133657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:01.133626 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" Nov 03 15:40:01.133657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:01.133640 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p" event={"ID":"928cb260-2bf9-4c3e-b72d-cf22412df2ff","Type":"ContainerDied","Data":"e976964ffec3e58c7ba9bf9e7bffc685015d1ebf2dbf3672de7e49e80507787d"} Nov 03 15:40:01.133890 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:01.133689 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e976964ffec3e58c7ba9bf9e7bffc685015d1ebf2dbf3672de7e49e80507787d" Nov 03 15:40:08.136527 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.136488 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h"] Nov 03 15:40:08.137640 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.137614 4290 cpu_manager.go:414] "containerMap: removing container" podUID="928cb260-2bf9-4c3e-b72d-cf22412df2ff" containerName="revoke-keys" Nov 03 15:40:08.137640 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.137641 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="928cb260-2bf9-4c3e-b72d-cf22412df2ff" containerName="revoke-keys" Nov 03 15:40:08.137911 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.137896 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="928cb260-2bf9-4c3e-b72d-cf22412df2ff" containerName="revoke-keys" Nov 03 15:40:08.142279 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.142252 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" Nov 03 15:40:08.197549 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.197513 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h"] Nov 03 15:40:08.240515 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.240459 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-90fcca\"" Nov 03 15:40:08.240844 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.240691 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-90fcca-dockercfg-8z8gx\"" Nov 03 15:40:08.327735 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.325001 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf"] Nov 03 15:40:08.335724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.332254 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" Nov 03 15:40:08.340044 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.339994 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp7pf\" (UniqueName: \"kubernetes.io/projected/8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705-kube-api-access-kp7pf\") pod \"payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h\" (UID: \"8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" Nov 03 15:40:08.340340 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.340274 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h\" (UID: \"8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" Nov 03 15:40:08.439856 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.439577 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf"] Nov 03 15:40:08.447848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.446819 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29f8x\" (UniqueName: \"kubernetes.io/projected/6e79fa4b-f118-4d97-a393-ad65d2c46cc6-kube-api-access-29f8x\") pod \"payload-processing-e2e-api-b-90fcca-b997c7df-6xctf\" (UID: \"6e79fa4b-f118-4d97-a393-ad65d2c46cc6\") " pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" Nov 03 15:40:08.447848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.447074 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e79fa4b-f118-4d97-a393-ad65d2c46cc6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-90fcca-b997c7df-6xctf\" (UID: \"6e79fa4b-f118-4d97-a393-ad65d2c46cc6\") " pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" Nov 03 15:40:08.447848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.447202 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp7pf\" (UniqueName: \"kubernetes.io/projected/8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705-kube-api-access-kp7pf\") pod \"payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h\" (UID: \"8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" Nov 03 15:40:08.447848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.447252 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h\" (UID: \"8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" Nov 03 15:40:08.448432 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.448381 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h\" (UID: \"8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" Nov 03 15:40:08.508872 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.508840 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp7pf\" (UniqueName: \"kubernetes.io/projected/8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705-kube-api-access-kp7pf\") pod \"payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h\" (UID: \"8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" Nov 03 15:40:08.548163 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.548122 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29f8x\" (UniqueName: \"kubernetes.io/projected/6e79fa4b-f118-4d97-a393-ad65d2c46cc6-kube-api-access-29f8x\") pod \"payload-processing-e2e-api-b-90fcca-b997c7df-6xctf\" (UID: \"6e79fa4b-f118-4d97-a393-ad65d2c46cc6\") " pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" Nov 03 15:40:08.548381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.548343 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e79fa4b-f118-4d97-a393-ad65d2c46cc6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-90fcca-b997c7df-6xctf\" (UID: \"6e79fa4b-f118-4d97-a393-ad65d2c46cc6\") " pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" Nov 03 15:40:08.549287 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.549260 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e79fa4b-f118-4d97-a393-ad65d2c46cc6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-90fcca-b997c7df-6xctf\" (UID: \"6e79fa4b-f118-4d97-a393-ad65d2c46cc6\") " pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" Nov 03 15:40:08.619020 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.618984 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29f8x\" (UniqueName: \"kubernetes.io/projected/6e79fa4b-f118-4d97-a393-ad65d2c46cc6-kube-api-access-29f8x\") pod \"payload-processing-e2e-api-b-90fcca-b997c7df-6xctf\" (UID: \"6e79fa4b-f118-4d97-a393-ad65d2c46cc6\") " pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" Nov 03 15:40:08.705878 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:08.705789 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:40:08.750727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.750659 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" Nov 03 15:40:08.756607 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:08.756578 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" Nov 03 15:40:09.160420 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:40:09.160388 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e79fa4b_f118_4d97_a393_ad65d2c46cc6.slice/crio-848a48a22c67e1144ad514c5c0a03abfdf3f64efba7b4bf34b47ebd9cbf6bd37 WatchSource:0}: Error finding container 848a48a22c67e1144ad514c5c0a03abfdf3f64efba7b4bf34b47ebd9cbf6bd37: Status 404 returned error can't find the container with id 848a48a22c67e1144ad514c5c0a03abfdf3f64efba7b4bf34b47ebd9cbf6bd37 Nov 03 15:40:09.162389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:09.162365 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf"] Nov 03 15:40:09.203562 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:09.203530 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" event={"ID":"6e79fa4b-f118-4d97-a393-ad65d2c46cc6","Type":"ContainerStarted","Data":"848a48a22c67e1144ad514c5c0a03abfdf3f64efba7b4bf34b47ebd9cbf6bd37"} Nov 03 15:40:09.212533 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:40:09.212501 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7a66a2_fd0c_4f32_9c24_c8e5a9b73705.slice/crio-f1ffb8b1c3e044483d0511bf889c60e69a062d04346b8a041b6507a10f44e136 WatchSource:0}: Error finding container f1ffb8b1c3e044483d0511bf889c60e69a062d04346b8a041b6507a10f44e136: Status 404 returned error can't find the container with id f1ffb8b1c3e044483d0511bf889c60e69a062d04346b8a041b6507a10f44e136 Nov 03 15:40:09.212897 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:09.212874 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h"] Nov 03 15:40:10.127546 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:10.127517 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" Nov 03 15:40:10.214253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:10.214205 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" event={"ID":"6e79fa4b-f118-4d97-a393-ad65d2c46cc6","Type":"ContainerStarted","Data":"3361c095289b146d7674f7febb333596add3af550024f98d09de932ac76383e0"} Nov 03 15:40:10.214756 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:10.214274 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" Nov 03 15:40:10.216087 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:10.216058 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" event={"ID":"8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705","Type":"ContainerStarted","Data":"67cae3c6688dca21ac410d6d4fd897a0717915c4c3623cefdadafea24f000aaf"} Nov 03 15:40:10.216219 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:10.216107 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" event={"ID":"8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705","Type":"ContainerStarted","Data":"f1ffb8b1c3e044483d0511bf889c60e69a062d04346b8a041b6507a10f44e136"} Nov 03 15:40:10.216219 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:10.216139 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" Nov 03 15:40:10.364377 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:10.364317 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" podStartSLOduration=2.364301733 podStartE2EDuration="2.364301733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:40:10.361636 +0000 UTC m=+1917.792593807" watchObservedRunningTime="2026-11-03 15:40:10.364301733 +0000 UTC m=+1917.795259581" Nov 03 15:40:10.495623 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:10.495495 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" podStartSLOduration=2.495470919 podStartE2EDuration="2.495470919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:40:10.488647748 +0000 UTC m=+1917.919605554" watchObservedRunningTime="2026-11-03 15:40:10.495470919 +0000 UTC m=+1917.926428723" Nov 03 15:40:11.373768 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:11.373728 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7310-c1b807"] Nov 03 15:40:11.379790 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:11.379761 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7310-c1b807" Nov 03 15:40:11.430891 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:11.430849 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7310-c1b807"] Nov 03 15:40:11.456527 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:11.456481 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj6sw\" (UniqueName: \"kubernetes.io/projected/f4442c20-e3f8-4897-928f-11e95685c262-kube-api-access-qj6sw\") pod \"test-curl-7310-c1b807\" (UID: \"f4442c20-e3f8-4897-928f-11e95685c262\") " pod="opendatahub/test-curl-7310-c1b807" Nov 03 15:40:11.475221 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:11.474810 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hb74t\"" Nov 03 15:40:11.557948 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:11.557910 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj6sw\" (UniqueName: \"kubernetes.io/projected/f4442c20-e3f8-4897-928f-11e95685c262-kube-api-access-qj6sw\") pod \"test-curl-7310-c1b807\" (UID: \"f4442c20-e3f8-4897-928f-11e95685c262\") " pod="opendatahub/test-curl-7310-c1b807" Nov 03 15:40:11.619734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:11.619673 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj6sw\" (UniqueName: \"kubernetes.io/projected/f4442c20-e3f8-4897-928f-11e95685c262-kube-api-access-qj6sw\") pod \"test-curl-7310-c1b807\" (UID: \"f4442c20-e3f8-4897-928f-11e95685c262\") " pod="opendatahub/test-curl-7310-c1b807" Nov 03 15:40:11.695833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:11.695744 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7310-c1b807" Nov 03 15:40:12.088574 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:40:12.088540 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4442c20_e3f8_4897_928f_11e95685c262.slice/crio-f65082d77c56255c14ed32cb48ee120ea86997b8a4f388942aa5774698c3e29f WatchSource:0}: Error finding container f65082d77c56255c14ed32cb48ee120ea86997b8a4f388942aa5774698c3e29f: Status 404 returned error can't find the container with id f65082d77c56255c14ed32cb48ee120ea86997b8a4f388942aa5774698c3e29f Nov 03 15:40:12.088912 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:12.088886 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7310-c1b807"] Nov 03 15:40:12.234094 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:12.234059 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7310-c1b807" event={"ID":"f4442c20-e3f8-4897-928f-11e95685c262","Type":"ContainerStarted","Data":"160314fb05b6f73485e948f7b5ef9c09b63c892f9c86448b5c1911d7c5f2dade"} Nov 03 15:40:12.234308 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:12.234105 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7310-c1b807" event={"ID":"f4442c20-e3f8-4897-928f-11e95685c262","Type":"ContainerStarted","Data":"f65082d77c56255c14ed32cb48ee120ea86997b8a4f388942aa5774698c3e29f"} Nov 03 15:40:12.349564 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:12.349456 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7310-c1b807" podStartSLOduration=1.349437738 podStartE2EDuration="1.349437738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:40:12.348477779 +0000 UTC m=+1919.779435586" watchObservedRunningTime="2026-11-03 15:40:12.349437738 +0000 UTC m=+1919.780395544" Nov 03 15:40:13.936418 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:13.936384 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7310-c1b807"] Nov 03 15:40:13.944171 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:13.944124 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7310-c1b807"] Nov 03 15:40:14.256752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:14.256635 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7310-c1b807" podUID="f4442c20-e3f8-4897-928f-11e95685c262" containerName="test-curl-7310-c1b807" containerID="cri-o://160314fb05b6f73485e948f7b5ef9c09b63c892f9c86448b5c1911d7c5f2dade" gracePeriod=2 Nov 03 15:40:14.308180 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:14.308144 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7310-c1b807\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="f4442c20-e3f8-4897-928f-11e95685c262" pod="opendatahub/test-curl-7310-c1b807" Nov 03 15:40:14.922329 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:14.922288 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj"] Nov 03 15:40:14.923089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:14.923074 4290 cpu_manager.go:414] "containerMap: removing container" podUID="f4442c20-e3f8-4897-928f-11e95685c262" containerName="test-curl-7310-c1b807" Nov 03 15:40:14.923150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:14.923091 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4442c20-e3f8-4897-928f-11e95685c262" containerName="test-curl-7310-c1b807" Nov 03 15:40:14.923227 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:14.923217 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4442c20-e3f8-4897-928f-11e95685c262" containerName="test-curl-7310-c1b807" Nov 03 15:40:14.927137 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:14.927121 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" Nov 03 15:40:15.000728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:14.996977 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj"] Nov 03 15:40:15.013730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:15.012111 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccrmh\" (UniqueName: \"kubernetes.io/projected/7b9cce47-d408-496c-ab69-58772c005366-kube-api-access-ccrmh\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj\" (UID: \"7b9cce47-d408-496c-ab69-58772c005366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" Nov 03 15:40:15.013730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:15.012654 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9cce47-d408-496c-ab69-58772c005366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj\" (UID: \"7b9cce47-d408-496c-ab69-58772c005366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" Nov 03 15:40:15.013730 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:15.012827 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9cce47-d408-496c-ab69-58772c005366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj\" (UID: \"7b9cce47-d408-496c-ab69-58772c005366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" Nov 03 15:40:15.034574 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:15.034537 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:40:15.034813 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:15.034715 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:40:15.114151 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:15.114112 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9cce47-d408-496c-ab69-58772c005366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj\" (UID: \"7b9cce47-d408-496c-ab69-58772c005366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" Nov 03 15:40:15.114356 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:15.114161 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9cce47-d408-496c-ab69-58772c005366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj\" (UID: \"7b9cce47-d408-496c-ab69-58772c005366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" Nov 03 15:40:15.114356 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:15.114288 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccrmh\" (UniqueName: \"kubernetes.io/projected/7b9cce47-d408-496c-ab69-58772c005366-kube-api-access-ccrmh\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj\" (UID: \"7b9cce47-d408-496c-ab69-58772c005366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" Nov 03 15:40:15.114854 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:15.114831 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9cce47-d408-496c-ab69-58772c005366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj\" (UID: \"7b9cce47-d408-496c-ab69-58772c005366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" Nov 03 15:40:15.116947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:15.116920 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9cce47-d408-496c-ab69-58772c005366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj\" (UID: \"7b9cce47-d408-496c-ab69-58772c005366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" Nov 03 15:40:15.178312 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:15.178232 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccrmh\" (UniqueName: \"kubernetes.io/projected/7b9cce47-d408-496c-ab69-58772c005366-kube-api-access-ccrmh\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj\" (UID: \"7b9cce47-d408-496c-ab69-58772c005366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" Nov 03 15:40:15.241865 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:15.241823 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" Nov 03 15:40:15.649811 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:15.649772 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj"] Nov 03 15:40:15.650284 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:40:15.650251 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9cce47_d408_496c_ab69_58772c005366.slice/crio-b93c95d18474f1334d83c40d91ae8fbd18e3d473edfdd8b26e0e551ccf4d2c51 WatchSource:0}: Error finding container b93c95d18474f1334d83c40d91ae8fbd18e3d473edfdd8b26e0e551ccf4d2c51: Status 404 returned error can't find the container with id b93c95d18474f1334d83c40d91ae8fbd18e3d473edfdd8b26e0e551ccf4d2c51 Nov 03 15:40:16.280379 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:16.280332 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" event={"ID":"7b9cce47-d408-496c-ab69-58772c005366","Type":"ContainerStarted","Data":"58bde94461003086ea5ac212a6c1c3c098599adb37cfe245923b666d5f51b70e"} Nov 03 15:40:16.280799 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:16.280409 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" event={"ID":"7b9cce47-d408-496c-ab69-58772c005366","Type":"ContainerStarted","Data":"b93c95d18474f1334d83c40d91ae8fbd18e3d473edfdd8b26e0e551ccf4d2c51"} Nov 03 15:40:16.406890 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:16.406827 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" podStartSLOduration=2.406807306 podStartE2EDuration="2.406807306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:40:16.400999179 +0000 UTC m=+1923.831956997" watchObservedRunningTime="2026-11-03 15:40:16.406807306 +0000 UTC m=+1923.837765112" Nov 03 15:40:16.502266 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:16.502232 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7310-c1b807" Nov 03 15:40:16.640669 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:16.640639 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4442c20-e3f8-4897-928f-11e95685c262-kube-api-access-qj6sw\" (UniqueName: \"kubernetes.io/projected/f4442c20-e3f8-4897-928f-11e95685c262-kube-api-access-qj6sw\") pod \"f4442c20-e3f8-4897-928f-11e95685c262\" (UID: \"f4442c20-e3f8-4897-928f-11e95685c262\") " Nov 03 15:40:16.643197 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:16.643119 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4442c20-e3f8-4897-928f-11e95685c262-kube-api-access-qj6sw" pod "f4442c20-e3f8-4897-928f-11e95685c262" (UID: "f4442c20-e3f8-4897-928f-11e95685c262"). InnerVolumeSpecName "kube-api-access-qj6sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:40:16.683562 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:16.683471 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4442c20-e3f8-4897-928f-11e95685c262" path="/var/lib/kubelet/pods/f4442c20-e3f8-4897-928f-11e95685c262/volumes" Nov 03 15:40:16.743243 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:16.743205 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj6sw\" (UniqueName: \"kubernetes.io/projected/f4442c20-e3f8-4897-928f-11e95685c262-kube-api-access-qj6sw\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:17.290166 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:17.290133 4290 generic.go:356] "Generic (PLEG): container finished" podID="7b9cce47-d408-496c-ab69-58772c005366" containerID="58bde94461003086ea5ac212a6c1c3c098599adb37cfe245923b666d5f51b70e" exitCode=0 Nov 03 15:40:17.290660 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:17.290222 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" event={"ID":"7b9cce47-d408-496c-ab69-58772c005366","Type":"ContainerDied","Data":"58bde94461003086ea5ac212a6c1c3c098599adb37cfe245923b666d5f51b70e"} Nov 03 15:40:17.291826 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:17.291795 4290 generic.go:356] "Generic (PLEG): container finished" podID="f4442c20-e3f8-4897-928f-11e95685c262" containerID="160314fb05b6f73485e948f7b5ef9c09b63c892f9c86448b5c1911d7c5f2dade" exitCode=137 Nov 03 15:40:17.291938 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:17.291842 4290 scope.go:122] "RemoveContainer" containerID="160314fb05b6f73485e948f7b5ef9c09b63c892f9c86448b5c1911d7c5f2dade" Nov 03 15:40:17.291938 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:17.291845 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7310-c1b807" Nov 03 15:40:17.302751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:17.302732 4290 scope.go:122] "RemoveContainer" containerID="160314fb05b6f73485e948f7b5ef9c09b63c892f9c86448b5c1911d7c5f2dade" Nov 03 15:40:17.302996 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:17.302978 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"160314fb05b6f73485e948f7b5ef9c09b63c892f9c86448b5c1911d7c5f2dade\": container with ID starting with 160314fb05b6f73485e948f7b5ef9c09b63c892f9c86448b5c1911d7c5f2dade not found: ID does not exist" containerID="160314fb05b6f73485e948f7b5ef9c09b63c892f9c86448b5c1911d7c5f2dade" Nov 03 15:40:17.303042 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:17.303003 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160314fb05b6f73485e948f7b5ef9c09b63c892f9c86448b5c1911d7c5f2dade"} err="failed to get container status \"160314fb05b6f73485e948f7b5ef9c09b63c892f9c86448b5c1911d7c5f2dade\": rpc error: code = NotFound desc = could not find container \"160314fb05b6f73485e948f7b5ef9c09b63c892f9c86448b5c1911d7c5f2dade\": container with ID starting with 160314fb05b6f73485e948f7b5ef9c09b63c892f9c86448b5c1911d7c5f2dade not found: ID does not exist" Nov 03 15:40:18.485108 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:18.485084 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" Nov 03 15:40:18.570920 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:18.570835 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9cce47-d408-496c-ab69-58772c005366-kube-api-access-ccrmh\" (UniqueName: \"kubernetes.io/projected/7b9cce47-d408-496c-ab69-58772c005366-kube-api-access-ccrmh\") pod \"7b9cce47-d408-496c-ab69-58772c005366\" (UID: \"7b9cce47-d408-496c-ab69-58772c005366\") " Nov 03 15:40:18.571072 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:18.570919 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b9cce47-d408-496c-ab69-58772c005366-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9cce47-d408-496c-ab69-58772c005366-maas-api-openshift-ca\") pod \"7b9cce47-d408-496c-ab69-58772c005366\" (UID: \"7b9cce47-d408-496c-ab69-58772c005366\") " Nov 03 15:40:18.571072 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:18.571015 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9cce47-d408-496c-ab69-58772c005366-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9cce47-d408-496c-ab69-58772c005366-maas-api-xks-ca\") pod \"7b9cce47-d408-496c-ab69-58772c005366\" (UID: \"7b9cce47-d408-496c-ab69-58772c005366\") " Nov 03 15:40:18.571329 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:18.571291 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9cce47-d408-496c-ab69-58772c005366-maas-api-openshift-ca" pod "7b9cce47-d408-496c-ab69-58772c005366" (UID: "7b9cce47-d408-496c-ab69-58772c005366"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:40:18.571854 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:18.571829 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9cce47-d408-496c-ab69-58772c005366-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:18.573281 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:18.573246 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9cce47-d408-496c-ab69-58772c005366-maas-api-xks-ca" pod "7b9cce47-d408-496c-ab69-58772c005366" (UID: "7b9cce47-d408-496c-ab69-58772c005366"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:40:18.573417 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:18.573394 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9cce47-d408-496c-ab69-58772c005366-kube-api-access-ccrmh" pod "7b9cce47-d408-496c-ab69-58772c005366" (UID: "7b9cce47-d408-496c-ab69-58772c005366"). InnerVolumeSpecName "kube-api-access-ccrmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:40:18.673080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:18.673025 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccrmh\" (UniqueName: \"kubernetes.io/projected/7b9cce47-d408-496c-ab69-58772c005366-kube-api-access-ccrmh\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:18.673080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:18.673062 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9cce47-d408-496c-ab69-58772c005366-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:19.313918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:19.313881 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" Nov 03 15:40:19.313918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:19.313899 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj" event={"ID":"7b9cce47-d408-496c-ab69-58772c005366","Type":"ContainerDied","Data":"b93c95d18474f1334d83c40d91ae8fbd18e3d473edfdd8b26e0e551ccf4d2c51"} Nov 03 15:40:19.314228 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:19.313964 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b93c95d18474f1334d83c40d91ae8fbd18e3d473edfdd8b26e0e551ccf4d2c51" Nov 03 15:40:21.226303 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:21.226272 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" Nov 03 15:40:21.226764 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:21.226333 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" Nov 03 15:40:26.496194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:26.496113 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl"] Nov 03 15:40:31.497609 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:31.497553 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" podUID="e8ba51c9-602b-47a6-bca9-5f9fa7547276" containerName="payload-processing" containerID="cri-o://6fc19d5d046165da0348bda2470167353008907a464f6b85d4979119b67c8db3" gracePeriod=25 Nov 03 15:40:31.908161 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:31.908126 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" podUID="e8ba51c9-602b-47a6-bca9-5f9fa7547276" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:40:38.704665 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:38.704627 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:40:38.770844 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:38.770813 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl"] Nov 03 15:40:38.771238 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:38.771211 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" containerID="cri-o://daf79fbd9b9595eed6dda3288851767c79a95f9798d46e7c8ef0991d7560b89c" gracePeriod=30 Nov 03 15:40:38.806288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:38.806256 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j"] Nov 03 15:40:38.806467 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:38.806447 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" podUID="fa7c2436-de33-42f9-b8c9-db8fc7d48229" containerName="istio-proxy" containerID="cri-o://bafcb72f3f88876a193cc4b0ab208c5b6393f610841c4d5506a9f31e40d2e3fd" gracePeriod=30 Nov 03 15:40:39.816443 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:39.816404 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7319-fc1bc3"] Nov 03 15:40:39.817165 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:39.817149 4290 cpu_manager.go:414] "containerMap: removing container" podUID="7b9cce47-d408-496c-ab69-58772c005366" containerName="revoke-keys" Nov 03 15:40:39.817211 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:39.817166 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9cce47-d408-496c-ab69-58772c005366" containerName="revoke-keys" Nov 03 15:40:39.817322 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:39.817312 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9cce47-d408-496c-ab69-58772c005366" containerName="revoke-keys" Nov 03 15:40:39.820954 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:39.820937 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-fc1bc3" Nov 03 15:40:39.828752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:39.828723 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss59h\" (UniqueName: \"kubernetes.io/projected/b4e53aad-57b5-41da-9847-2bd1b1649112-kube-api-access-ss59h\") pod \"test-curl-7319-fc1bc3\" (UID: \"b4e53aad-57b5-41da-9847-2bd1b1649112\") " pod="opendatahub/test-curl-7319-fc1bc3" Nov 03 15:40:39.882599 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:39.882560 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7319-fc1bc3"] Nov 03 15:40:39.921642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:39.921617 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hb74t\"" Nov 03 15:40:39.930646 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:39.930615 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss59h\" (UniqueName: \"kubernetes.io/projected/b4e53aad-57b5-41da-9847-2bd1b1649112-kube-api-access-ss59h\") pod \"test-curl-7319-fc1bc3\" (UID: \"b4e53aad-57b5-41da-9847-2bd1b1649112\") " pod="opendatahub/test-curl-7319-fc1bc3" Nov 03 15:40:39.994246 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:39.994209 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss59h\" (UniqueName: \"kubernetes.io/projected/b4e53aad-57b5-41da-9847-2bd1b1649112-kube-api-access-ss59h\") pod \"test-curl-7319-fc1bc3\" (UID: \"b4e53aad-57b5-41da-9847-2bd1b1649112\") " pod="opendatahub/test-curl-7319-fc1bc3" Nov 03 15:40:40.046330 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:40.046297 4290 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:40:40.046500 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:40.046371 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" podUID="fa7c2436-de33-42f9-b8c9-db8fc7d48229" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:40:40.134076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:40.134050 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-fc1bc3" Nov 03 15:40:40.530301 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:40.530273 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7319-fc1bc3"] Nov 03 15:40:40.530427 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:40:40.530378 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4e53aad_57b5_41da_9847_2bd1b1649112.slice/crio-dc26c97aaf42c503fb14e5ef94e4b4ba77deb7a73057463204282280acd63fde WatchSource:0}: Error finding container dc26c97aaf42c503fb14e5ef94e4b4ba77deb7a73057463204282280acd63fde: Status 404 returned error can't find the container with id dc26c97aaf42c503fb14e5ef94e4b4ba77deb7a73057463204282280acd63fde Nov 03 15:40:40.575985 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:40.575945 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7319-fc1bc3" event={"ID":"b4e53aad-57b5-41da-9847-2bd1b1649112","Type":"ContainerStarted","Data":"dc26c97aaf42c503fb14e5ef94e4b4ba77deb7a73057463204282280acd63fde"} Nov 03 15:40:41.586027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:41.585990 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7319-fc1bc3" event={"ID":"b4e53aad-57b5-41da-9847-2bd1b1649112","Type":"ContainerStarted","Data":"f250b6ee59a8fb05c0e10614d5784163eefff29d8f52b5392c8d12a54134bb3e"} Nov 03 15:40:41.703005 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:41.702956 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7319-fc1bc3" podStartSLOduration=2.702939591 podStartE2EDuration="2.702939591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:40:41.700121466 +0000 UTC m=+1949.131079277" watchObservedRunningTime="2026-11-03 15:40:41.702939591 +0000 UTC m=+1949.133897397" Nov 03 15:40:41.907960 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:41.907925 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" podUID="e8ba51c9-602b-47a6-bca9-5f9fa7547276" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:40:43.161905 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:43.161858 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7319-fc1bc3"] Nov 03 15:40:43.167503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:43.167476 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7319-fc1bc3"] Nov 03 15:40:43.323750 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:43.323718 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7319-c50649"] Nov 03 15:40:43.324479 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:43.324465 4290 cpu_manager.go:414] "containerMap: removing container" podUID="b4e53aad-57b5-41da-9847-2bd1b1649112" containerName="test-curl-7319-fc1bc3" Nov 03 15:40:43.324521 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:43.324481 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e53aad-57b5-41da-9847-2bd1b1649112" containerName="test-curl-7319-fc1bc3" Nov 03 15:40:43.324599 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:43.324590 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e53aad-57b5-41da-9847-2bd1b1649112" containerName="test-curl-7319-fc1bc3" Nov 03 15:40:43.328498 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:43.328481 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-c50649" Nov 03 15:40:43.384862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:43.384831 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7319-c50649"] Nov 03 15:40:43.385035 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:43.384976 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552sq\" (UniqueName: \"kubernetes.io/projected/912c5274-1315-415c-a9f8-9842a5bc27a0-kube-api-access-552sq\") pod \"test-curl-7319-c50649\" (UID: \"912c5274-1315-415c-a9f8-9842a5bc27a0\") " pod="opendatahub/test-curl-7319-c50649" Nov 03 15:40:43.486423 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:43.486349 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552sq\" (UniqueName: \"kubernetes.io/projected/912c5274-1315-415c-a9f8-9842a5bc27a0-kube-api-access-552sq\") pod \"test-curl-7319-c50649\" (UID: \"912c5274-1315-415c-a9f8-9842a5bc27a0\") " pod="opendatahub/test-curl-7319-c50649" Nov 03 15:40:43.547926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:43.547897 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-552sq\" (UniqueName: \"kubernetes.io/projected/912c5274-1315-415c-a9f8-9842a5bc27a0-kube-api-access-552sq\") pod \"test-curl-7319-c50649\" (UID: \"912c5274-1315-415c-a9f8-9842a5bc27a0\") " pod="opendatahub/test-curl-7319-c50649" Nov 03 15:40:43.603549 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:43.603513 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7319-fc1bc3" podUID="b4e53aad-57b5-41da-9847-2bd1b1649112" containerName="test-curl-7319-fc1bc3" containerID="cri-o://f250b6ee59a8fb05c0e10614d5784163eefff29d8f52b5392c8d12a54134bb3e" gracePeriod=2 Nov 03 15:40:43.641459 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:43.641432 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-c50649" Nov 03 15:40:43.654974 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:43.654943 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-fc1bc3\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="b4e53aad-57b5-41da-9847-2bd1b1649112" pod="opendatahub/test-curl-7319-fc1bc3" Nov 03 15:40:44.034025 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.033989 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7319-c50649"] Nov 03 15:40:44.039667 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:40:44.039635 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod912c5274_1315_415c_a9f8_9842a5bc27a0.slice/crio-96d05871d6f0f5b9b4ba4baf0d2916e5c8ac9fb34a24e61e0fe7ec3c55560e04 WatchSource:0}: Error finding container 96d05871d6f0f5b9b4ba4baf0d2916e5c8ac9fb34a24e61e0fe7ec3c55560e04: Status 404 returned error can't find the container with id 96d05871d6f0f5b9b4ba4baf0d2916e5c8ac9fb34a24e61e0fe7ec3c55560e04 Nov 03 15:40:44.073253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.073217 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:40:44.126041 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.126020 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:40:44.196968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.196934 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-workload-certs\") pod \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " Nov 03 15:40:44.196968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.196972 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-data\") pod \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " Nov 03 15:40:44.197435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197005 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-envoy\") pod \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " Nov 03 15:40:44.197435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197056 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-credential-socket\") pod \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " Nov 03 15:40:44.197435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197100 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-token\" (UniqueName: \"kubernetes.io/projected/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-token\") pod \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " Nov 03 15:40:44.197435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197130 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istiod-ca-cert\") pod \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " Nov 03 15:40:44.197435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197161 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-envoy\") pod \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " Nov 03 15:40:44.197435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197202 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-data\") pod \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " Nov 03 15:40:44.197435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197232 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-data" pod "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" (UID: "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:40:44.197435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197247 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-workload-certs\") pod \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " Nov 03 15:40:44.197435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197245 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-workload-certs" pod "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" (UID: "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:40:44.197435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197347 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-credential-socket" pod "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" (UID: "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:40:44.197435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197396 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-workload-socket\") pod \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " Nov 03 15:40:44.197435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197431 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-token\" (UniqueName: \"kubernetes.io/projected/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-token\") pod \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " Nov 03 15:40:44.198111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197465 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-kube-api-access-swkzb\" (UniqueName: \"kubernetes.io/projected/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-kube-api-access-swkzb\") pod \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " Nov 03 15:40:44.198111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197522 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7c2436-de33-42f9-b8c9-db8fc7d48229-kube-api-access-85875\" (UniqueName: \"kubernetes.io/projected/fa7c2436-de33-42f9-b8c9-db8fc7d48229-kube-api-access-85875\") pod \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " Nov 03 15:40:44.198111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197555 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-podinfo\") pod \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " Nov 03 15:40:44.198111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197564 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-data" pod "fa7c2436-de33-42f9-b8c9-db8fc7d48229" (UID: "fa7c2436-de33-42f9-b8c9-db8fc7d48229"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:40:44.198111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197608 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istiod-ca-cert" pod "fa7c2436-de33-42f9-b8c9-db8fc7d48229" (UID: "fa7c2436-de33-42f9-b8c9-db8fc7d48229"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:40:44.198111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197621 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-podinfo\") pod \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " Nov 03 15:40:44.198111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197668 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-workload-socket\") pod \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " Nov 03 15:40:44.198111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197734 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-credential-socket\") pod \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\" (UID: \"fa7c2436-de33-42f9-b8c9-db8fc7d48229\") " Nov 03 15:40:44.198111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.197893 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istiod-ca-cert\") pod \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\" (UID: \"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f\") " Nov 03 15:40:44.198578 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.198337 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-workload-certs" pod "fa7c2436-de33-42f9-b8c9-db8fc7d48229" (UID: "fa7c2436-de33-42f9-b8c9-db8fc7d48229"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:40:44.198578 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.198515 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-workload-socket" pod "fa7c2436-de33-42f9-b8c9-db8fc7d48229" (UID: "fa7c2436-de33-42f9-b8c9-db8fc7d48229"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:40:44.198883 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.198854 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istiod-ca-cert" pod "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" (UID: "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:40:44.199391 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.199366 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.199522 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.199506 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.199633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.199608 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.199633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.199636 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.199852 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.199650 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.199852 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.199665 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.199852 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.199680 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.199852 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.199694 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.200118 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.200094 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-credential-socket" pod "fa7c2436-de33-42f9-b8c9-db8fc7d48229" (UID: "fa7c2436-de33-42f9-b8c9-db8fc7d48229"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:40:44.200284 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.200260 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-workload-socket" pod "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" (UID: "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:40:44.200545 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.200520 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-envoy" pod "fa7c2436-de33-42f9-b8c9-db8fc7d48229" (UID: "fa7c2436-de33-42f9-b8c9-db8fc7d48229"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:40:44.200680 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.200644 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-token" pod "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" (UID: "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:40:44.200882 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.200858 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-envoy" pod "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" (UID: "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:40:44.200964 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.200884 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7c2436-de33-42f9-b8c9-db8fc7d48229-kube-api-access-85875" pod "fa7c2436-de33-42f9-b8c9-db8fc7d48229" (UID: "fa7c2436-de33-42f9-b8c9-db8fc7d48229"). InnerVolumeSpecName "kube-api-access-85875". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:40:44.200964 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.200932 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-token" pod "fa7c2436-de33-42f9-b8c9-db8fc7d48229" (UID: "fa7c2436-de33-42f9-b8c9-db8fc7d48229"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:40:44.201103 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.201087 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-kube-api-access-swkzb" pod "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" (UID: "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f"). InnerVolumeSpecName "kube-api-access-swkzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:40:44.201565 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.201547 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-podinfo" pod "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" (UID: "bb3b55d8-d384-478a-acb8-9ebdd4b06b2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:40:44.202362 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.202344 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-podinfo" pod "fa7c2436-de33-42f9-b8c9-db8fc7d48229" (UID: "fa7c2436-de33-42f9-b8c9-db8fc7d48229"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:40:44.264835 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:44.264793 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-fc1bc3\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="b4e53aad-57b5-41da-9847-2bd1b1649112" pod="opendatahub/test-curl-7319-fc1bc3" Nov 03 15:40:44.301064 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.301033 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.301064 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.301061 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swkzb\" (UniqueName: \"kubernetes.io/projected/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-kube-api-access-swkzb\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.301256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.301072 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85875\" (UniqueName: \"kubernetes.io/projected/fa7c2436-de33-42f9-b8c9-db8fc7d48229-kube-api-access-85875\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.301256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.301081 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.301256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.301102 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.301256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.301111 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.301256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.301119 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.301256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.301128 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7c2436-de33-42f9-b8c9-db8fc7d48229-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.301256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.301136 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.301256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.301144 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:44.442195 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:44.442157 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-fc1bc3\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="b4e53aad-57b5-41da-9847-2bd1b1649112" pod="opendatahub/test-curl-7319-fc1bc3" Nov 03 15:40:44.614039 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.614002 4290 generic.go:356] "Generic (PLEG): container finished" podID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerID="daf79fbd9b9595eed6dda3288851767c79a95f9798d46e7c8ef0991d7560b89c" exitCode=0 Nov 03 15:40:44.614216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.614059 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" Nov 03 15:40:44.614216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.614098 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" event={"ID":"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f","Type":"ContainerDied","Data":"daf79fbd9b9595eed6dda3288851767c79a95f9798d46e7c8ef0991d7560b89c"} Nov 03 15:40:44.614216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.614193 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl" event={"ID":"bb3b55d8-d384-478a-acb8-9ebdd4b06b2f","Type":"ContainerDied","Data":"d087da7adc371bbb41fa66d50ef5c0d61f563d3da2de74d4a0122c56fc25472f"} Nov 03 15:40:44.614369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.614222 4290 scope.go:122] "RemoveContainer" containerID="daf79fbd9b9595eed6dda3288851767c79a95f9798d46e7c8ef0991d7560b89c" Nov 03 15:40:44.615985 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.615949 4290 generic.go:356] "Generic (PLEG): container finished" podID="fa7c2436-de33-42f9-b8c9-db8fc7d48229" containerID="bafcb72f3f88876a193cc4b0ab208c5b6393f610841c4d5506a9f31e40d2e3fd" exitCode=0 Nov 03 15:40:44.616291 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.616271 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" Nov 03 15:40:44.616460 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.616263 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" event={"ID":"fa7c2436-de33-42f9-b8c9-db8fc7d48229","Type":"ContainerDied","Data":"bafcb72f3f88876a193cc4b0ab208c5b6393f610841c4d5506a9f31e40d2e3fd"} Nov 03 15:40:44.616579 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.616499 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j" event={"ID":"fa7c2436-de33-42f9-b8c9-db8fc7d48229","Type":"ContainerDied","Data":"f60dada3c8c24074f52529d7897fd0522b02f16a27de15bd114687290183aa92"} Nov 03 15:40:44.618316 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.618291 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7319-c50649" event={"ID":"912c5274-1315-415c-a9f8-9842a5bc27a0","Type":"ContainerStarted","Data":"c0778b2fe418b346ce2e8b441da2b31298b0f98e347ebff4684f0d3562d45f29"} Nov 03 15:40:44.618425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.618326 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7319-c50649" event={"ID":"912c5274-1315-415c-a9f8-9842a5bc27a0","Type":"ContainerStarted","Data":"96d05871d6f0f5b9b4ba4baf0d2916e5c8ac9fb34a24e61e0fe7ec3c55560e04"} Nov 03 15:40:44.632179 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.632104 4290 scope.go:122] "RemoveContainer" containerID="daf79fbd9b9595eed6dda3288851767c79a95f9798d46e7c8ef0991d7560b89c" Nov 03 15:40:44.632636 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:44.632606 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf79fbd9b9595eed6dda3288851767c79a95f9798d46e7c8ef0991d7560b89c\": container with ID starting with daf79fbd9b9595eed6dda3288851767c79a95f9798d46e7c8ef0991d7560b89c not found: ID does not exist" containerID="daf79fbd9b9595eed6dda3288851767c79a95f9798d46e7c8ef0991d7560b89c" Nov 03 15:40:44.632769 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.632647 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf79fbd9b9595eed6dda3288851767c79a95f9798d46e7c8ef0991d7560b89c"} err="failed to get container status \"daf79fbd9b9595eed6dda3288851767c79a95f9798d46e7c8ef0991d7560b89c\": rpc error: code = NotFound desc = could not find container \"daf79fbd9b9595eed6dda3288851767c79a95f9798d46e7c8ef0991d7560b89c\": container with ID starting with daf79fbd9b9595eed6dda3288851767c79a95f9798d46e7c8ef0991d7560b89c not found: ID does not exist" Nov 03 15:40:44.632769 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.632673 4290 scope.go:122] "RemoveContainer" containerID="bafcb72f3f88876a193cc4b0ab208c5b6393f610841c4d5506a9f31e40d2e3fd" Nov 03 15:40:44.644167 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.644149 4290 scope.go:122] "RemoveContainer" containerID="bafcb72f3f88876a193cc4b0ab208c5b6393f610841c4d5506a9f31e40d2e3fd" Nov 03 15:40:44.644419 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:44.644401 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bafcb72f3f88876a193cc4b0ab208c5b6393f610841c4d5506a9f31e40d2e3fd\": container with ID starting with bafcb72f3f88876a193cc4b0ab208c5b6393f610841c4d5506a9f31e40d2e3fd not found: ID does not exist" containerID="bafcb72f3f88876a193cc4b0ab208c5b6393f610841c4d5506a9f31e40d2e3fd" Nov 03 15:40:44.644470 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.644426 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafcb72f3f88876a193cc4b0ab208c5b6393f610841c4d5506a9f31e40d2e3fd"} err="failed to get container status \"bafcb72f3f88876a193cc4b0ab208c5b6393f610841c4d5506a9f31e40d2e3fd\": rpc error: code = NotFound desc = could not find container \"bafcb72f3f88876a193cc4b0ab208c5b6393f610841c4d5506a9f31e40d2e3fd\": container with ID starting with bafcb72f3f88876a193cc4b0ab208c5b6393f610841c4d5506a9f31e40d2e3fd not found: ID does not exist" Nov 03 15:40:44.665626 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:44.665600 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-fc1bc3\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="b4e53aad-57b5-41da-9847-2bd1b1649112" pod="opendatahub/test-curl-7319-fc1bc3" Nov 03 15:40:44.835761 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.835651 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j"] Nov 03 15:40:44.841475 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.841450 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-trg4j"] Nov 03 15:40:44.955485 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:44.955430 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7319-c50649" podStartSLOduration=1.9554161300000001 podStartE2EDuration="1.95541613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:40:44.952952551 +0000 UTC m=+1952.383910358" watchObservedRunningTime="2026-11-03 15:40:44.95541613 +0000 UTC m=+1952.386373936" Nov 03 15:40:45.069808 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:45.069775 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl"] Nov 03 15:40:45.074934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:45.074911 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-xz5pl"] Nov 03 15:40:45.885099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:45.884879 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-fc1bc3" Nov 03 15:40:46.025157 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.025122 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e53aad-57b5-41da-9847-2bd1b1649112-kube-api-access-ss59h\" (UniqueName: \"kubernetes.io/projected/b4e53aad-57b5-41da-9847-2bd1b1649112-kube-api-access-ss59h\") pod \"b4e53aad-57b5-41da-9847-2bd1b1649112\" (UID: \"b4e53aad-57b5-41da-9847-2bd1b1649112\") " Nov 03 15:40:46.027513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.027475 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e53aad-57b5-41da-9847-2bd1b1649112-kube-api-access-ss59h" pod "b4e53aad-57b5-41da-9847-2bd1b1649112" (UID: "b4e53aad-57b5-41da-9847-2bd1b1649112"). InnerVolumeSpecName "kube-api-access-ss59h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:40:46.127817 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.127779 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss59h\" (UniqueName: \"kubernetes.io/projected/b4e53aad-57b5-41da-9847-2bd1b1649112-kube-api-access-ss59h\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:46.630802 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.630671 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7319-c50649"] Nov 03 15:40:46.642748 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.638311 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7319-c50649"] Nov 03 15:40:46.645715 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.645661 4290 generic.go:356] "Generic (PLEG): container finished" podID="b4e53aad-57b5-41da-9847-2bd1b1649112" containerID="f250b6ee59a8fb05c0e10614d5784163eefff29d8f52b5392c8d12a54134bb3e" exitCode=137 Nov 03 15:40:46.645864 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.645743 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-fc1bc3" Nov 03 15:40:46.645864 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.645749 4290 scope.go:122] "RemoveContainer" containerID="f250b6ee59a8fb05c0e10614d5784163eefff29d8f52b5392c8d12a54134bb3e" Nov 03 15:40:46.646164 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.646109 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7319-c50649" podUID="912c5274-1315-415c-a9f8-9842a5bc27a0" containerName="test-curl-7319-c50649" containerID="cri-o://c0778b2fe418b346ce2e8b441da2b31298b0f98e347ebff4684f0d3562d45f29" gracePeriod=2 Nov 03 15:40:46.662326 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.662307 4290 scope.go:122] "RemoveContainer" containerID="f250b6ee59a8fb05c0e10614d5784163eefff29d8f52b5392c8d12a54134bb3e" Nov 03 15:40:46.662634 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:46.662613 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f250b6ee59a8fb05c0e10614d5784163eefff29d8f52b5392c8d12a54134bb3e\": container with ID starting with f250b6ee59a8fb05c0e10614d5784163eefff29d8f52b5392c8d12a54134bb3e not found: ID does not exist" containerID="f250b6ee59a8fb05c0e10614d5784163eefff29d8f52b5392c8d12a54134bb3e" Nov 03 15:40:46.662760 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.662643 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f250b6ee59a8fb05c0e10614d5784163eefff29d8f52b5392c8d12a54134bb3e"} err="failed to get container status \"f250b6ee59a8fb05c0e10614d5784163eefff29d8f52b5392c8d12a54134bb3e\": rpc error: code = NotFound desc = could not find container \"f250b6ee59a8fb05c0e10614d5784163eefff29d8f52b5392c8d12a54134bb3e\": container with ID starting with f250b6ee59a8fb05c0e10614d5784163eefff29d8f52b5392c8d12a54134bb3e not found: ID does not exist" Nov 03 15:40:46.689537 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.689504 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e53aad-57b5-41da-9847-2bd1b1649112" path="/var/lib/kubelet/pods/b4e53aad-57b5-41da-9847-2bd1b1649112/volumes" Nov 03 15:40:46.690326 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.690293 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" path="/var/lib/kubelet/pods/bb3b55d8-d384-478a-acb8-9ebdd4b06b2f/volumes" Nov 03 15:40:46.691000 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.690977 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa7c2436-de33-42f9-b8c9-db8fc7d48229" path="/var/lib/kubelet/pods/fa7c2436-de33-42f9-b8c9-db8fc7d48229/volumes" Nov 03 15:40:46.699120 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:46.697519 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-c50649\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="912c5274-1315-415c-a9f8-9842a5bc27a0" pod="opendatahub/test-curl-7319-c50649" Nov 03 15:40:46.796154 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.796118 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7319-1b0910"] Nov 03 15:40:46.797194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.797165 4290 cpu_manager.go:414] "containerMap: removing container" podUID="912c5274-1315-415c-a9f8-9842a5bc27a0" containerName="test-curl-7319-c50649" Nov 03 15:40:46.797194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.797190 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912c5274-1315-415c-a9f8-9842a5bc27a0" containerName="test-curl-7319-c50649" Nov 03 15:40:46.797386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.797247 4290 cpu_manager.go:414] "containerMap: removing container" podUID="fa7c2436-de33-42f9-b8c9-db8fc7d48229" containerName="istio-proxy" Nov 03 15:40:46.797386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.797255 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7c2436-de33-42f9-b8c9-db8fc7d48229" containerName="istio-proxy" Nov 03 15:40:46.797386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.797264 4290 cpu_manager.go:414] "containerMap: removing container" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" Nov 03 15:40:46.797386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.797271 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" Nov 03 15:40:46.797577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.797416 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7c2436-de33-42f9-b8c9-db8fc7d48229" containerName="istio-proxy" Nov 03 15:40:46.797577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.797430 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3b55d8-d384-478a-acb8-9ebdd4b06b2f" containerName="istio-proxy" Nov 03 15:40:46.797577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.797445 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="912c5274-1315-415c-a9f8-9842a5bc27a0" containerName="test-curl-7319-c50649" Nov 03 15:40:46.803288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.803266 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-1b0910" Nov 03 15:40:46.854230 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.853912 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7319-1b0910"] Nov 03 15:40:46.940221 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:46.940122 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrpfs\" (UniqueName: \"kubernetes.io/projected/996ff5c7-5d26-4fae-8b03-34c8ef551ff9-kube-api-access-nrpfs\") pod \"test-curl-7319-1b0910\" (UID: \"996ff5c7-5d26-4fae-8b03-34c8ef551ff9\") " pod="opendatahub/test-curl-7319-1b0910" Nov 03 15:40:47.044377 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:47.044342 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrpfs\" (UniqueName: \"kubernetes.io/projected/996ff5c7-5d26-4fae-8b03-34c8ef551ff9-kube-api-access-nrpfs\") pod \"test-curl-7319-1b0910\" (UID: \"996ff5c7-5d26-4fae-8b03-34c8ef551ff9\") " pod="opendatahub/test-curl-7319-1b0910" Nov 03 15:40:47.111293 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:47.111257 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrpfs\" (UniqueName: \"kubernetes.io/projected/996ff5c7-5d26-4fae-8b03-34c8ef551ff9-kube-api-access-nrpfs\") pod \"test-curl-7319-1b0910\" (UID: \"996ff5c7-5d26-4fae-8b03-34c8ef551ff9\") " pod="opendatahub/test-curl-7319-1b0910" Nov 03 15:40:47.117392 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:47.117360 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-1b0910" Nov 03 15:40:47.517133 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:40:47.517101 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod996ff5c7_5d26_4fae_8b03_34c8ef551ff9.slice/crio-3a8fd5538f196d5a542806f0ffde8c5a628bf5d754e26bafadea23ebbb1d57d7 WatchSource:0}: Error finding container 3a8fd5538f196d5a542806f0ffde8c5a628bf5d754e26bafadea23ebbb1d57d7: Status 404 returned error can't find the container with id 3a8fd5538f196d5a542806f0ffde8c5a628bf5d754e26bafadea23ebbb1d57d7 Nov 03 15:40:47.517427 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:47.517337 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7319-1b0910"] Nov 03 15:40:47.657890 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:47.657856 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7319-1b0910" event={"ID":"996ff5c7-5d26-4fae-8b03-34c8ef551ff9","Type":"ContainerStarted","Data":"8c1013e951e8c09902b510cff12695d7e1daec439feee4e7d0a93d79598f27c6"} Nov 03 15:40:47.658047 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:47.657902 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7319-1b0910" event={"ID":"996ff5c7-5d26-4fae-8b03-34c8ef551ff9","Type":"ContainerStarted","Data":"3a8fd5538f196d5a542806f0ffde8c5a628bf5d754e26bafadea23ebbb1d57d7"} Nov 03 15:40:47.782155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:47.782027 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7319-1b0910" podStartSLOduration=1.782009462 podStartE2EDuration="1.782009462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:40:47.773796954 +0000 UTC m=+1955.204754760" watchObservedRunningTime="2026-11-03 15:40:47.782009462 +0000 UTC m=+1955.212967269" Nov 03 15:40:49.053137 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.053108 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-c50649" Nov 03 15:40:49.099784 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.099752 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912c5274-1315-415c-a9f8-9842a5bc27a0-kube-api-access-552sq\" (UniqueName: \"kubernetes.io/projected/912c5274-1315-415c-a9f8-9842a5bc27a0-kube-api-access-552sq\") pod \"912c5274-1315-415c-a9f8-9842a5bc27a0\" (UID: \"912c5274-1315-415c-a9f8-9842a5bc27a0\") " Nov 03 15:40:49.102219 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.102194 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912c5274-1315-415c-a9f8-9842a5bc27a0-kube-api-access-552sq" pod "912c5274-1315-415c-a9f8-9842a5bc27a0" (UID: "912c5274-1315-415c-a9f8-9842a5bc27a0"). InnerVolumeSpecName "kube-api-access-552sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:40:49.201942 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.201871 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-552sq\" (UniqueName: \"kubernetes.io/projected/912c5274-1315-415c-a9f8-9842a5bc27a0-kube-api-access-552sq\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:49.306687 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.306651 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7319-1b0910"] Nov 03 15:40:49.311580 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.311550 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7319-1b0910"] Nov 03 15:40:49.467562 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.467478 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7319-77b693"] Nov 03 15:40:49.468358 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.468334 4290 cpu_manager.go:414] "containerMap: removing container" podUID="996ff5c7-5d26-4fae-8b03-34c8ef551ff9" containerName="test-curl-7319-1b0910" Nov 03 15:40:49.468358 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.468357 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996ff5c7-5d26-4fae-8b03-34c8ef551ff9" containerName="test-curl-7319-1b0910" Nov 03 15:40:49.468547 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.468532 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="996ff5c7-5d26-4fae-8b03-34c8ef551ff9" containerName="test-curl-7319-1b0910" Nov 03 15:40:49.472291 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.472272 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-77b693" Nov 03 15:40:49.506573 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.506548 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjx7l\" (UniqueName: \"kubernetes.io/projected/8647e935-6cbe-4c09-abad-60e2b2a6a72d-kube-api-access-hjx7l\") pod \"test-curl-7319-77b693\" (UID: \"8647e935-6cbe-4c09-abad-60e2b2a6a72d\") " pod="opendatahub/test-curl-7319-77b693" Nov 03 15:40:49.527681 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.527655 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7319-77b693"] Nov 03 15:40:49.607586 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.607556 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjx7l\" (UniqueName: \"kubernetes.io/projected/8647e935-6cbe-4c09-abad-60e2b2a6a72d-kube-api-access-hjx7l\") pod \"test-curl-7319-77b693\" (UID: \"8647e935-6cbe-4c09-abad-60e2b2a6a72d\") " pod="opendatahub/test-curl-7319-77b693" Nov 03 15:40:49.670464 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.670428 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjx7l\" (UniqueName: \"kubernetes.io/projected/8647e935-6cbe-4c09-abad-60e2b2a6a72d-kube-api-access-hjx7l\") pod \"test-curl-7319-77b693\" (UID: \"8647e935-6cbe-4c09-abad-60e2b2a6a72d\") " pod="opendatahub/test-curl-7319-77b693" Nov 03 15:40:49.728337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.728247 4290 generic.go:356] "Generic (PLEG): container finished" podID="912c5274-1315-415c-a9f8-9842a5bc27a0" containerID="c0778b2fe418b346ce2e8b441da2b31298b0f98e347ebff4684f0d3562d45f29" exitCode=137 Nov 03 15:40:49.728337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.728292 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-c50649" Nov 03 15:40:49.728548 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.728346 4290 scope.go:122] "RemoveContainer" containerID="c0778b2fe418b346ce2e8b441da2b31298b0f98e347ebff4684f0d3562d45f29" Nov 03 15:40:49.728600 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.728583 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7319-1b0910" podUID="996ff5c7-5d26-4fae-8b03-34c8ef551ff9" containerName="test-curl-7319-1b0910" containerID="cri-o://8c1013e951e8c09902b510cff12695d7e1daec439feee4e7d0a93d79598f27c6" gracePeriod=2 Nov 03 15:40:49.742833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.742808 4290 scope.go:122] "RemoveContainer" containerID="c0778b2fe418b346ce2e8b441da2b31298b0f98e347ebff4684f0d3562d45f29" Nov 03 15:40:49.743155 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:49.743134 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0778b2fe418b346ce2e8b441da2b31298b0f98e347ebff4684f0d3562d45f29\": container with ID starting with c0778b2fe418b346ce2e8b441da2b31298b0f98e347ebff4684f0d3562d45f29 not found: ID does not exist" containerID="c0778b2fe418b346ce2e8b441da2b31298b0f98e347ebff4684f0d3562d45f29" Nov 03 15:40:49.743238 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.743164 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0778b2fe418b346ce2e8b441da2b31298b0f98e347ebff4684f0d3562d45f29"} err="failed to get container status \"c0778b2fe418b346ce2e8b441da2b31298b0f98e347ebff4684f0d3562d45f29\": rpc error: code = NotFound desc = could not find container \"c0778b2fe418b346ce2e8b441da2b31298b0f98e347ebff4684f0d3562d45f29\": container with ID starting with c0778b2fe418b346ce2e8b441da2b31298b0f98e347ebff4684f0d3562d45f29 not found: ID does not exist" Nov 03 15:40:49.779585 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:49.779556 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-1b0910\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="996ff5c7-5d26-4fae-8b03-34c8ef551ff9" pod="opendatahub/test-curl-7319-1b0910" Nov 03 15:40:49.788897 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:49.788877 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-77b693" Nov 03 15:40:49.831241 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:49.831198 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-1b0910\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="996ff5c7-5d26-4fae-8b03-34c8ef551ff9" pod="opendatahub/test-curl-7319-1b0910" Nov 03 15:40:49.885114 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:49.883539 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-c50649\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="912c5274-1315-415c-a9f8-9842a5bc27a0" pod="opendatahub/test-curl-7319-c50649" Nov 03 15:40:50.178353 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:40:50.178311 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8647e935_6cbe_4c09_abad_60e2b2a6a72d.slice/crio-0592f0c633d4388b4457d68466c8a9e4adfc90b69de275ed904d087a770b2355 WatchSource:0}: Error finding container 0592f0c633d4388b4457d68466c8a9e4adfc90b69de275ed904d087a770b2355: Status 404 returned error can't find the container with id 0592f0c633d4388b4457d68466c8a9e4adfc90b69de275ed904d087a770b2355 Nov 03 15:40:50.179026 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:50.179001 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7319-77b693"] Nov 03 15:40:50.683314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:50.683281 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="912c5274-1315-415c-a9f8-9842a5bc27a0" path="/var/lib/kubelet/pods/912c5274-1315-415c-a9f8-9842a5bc27a0/volumes" Nov 03 15:40:50.741600 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:50.741553 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7319-77b693" event={"ID":"8647e935-6cbe-4c09-abad-60e2b2a6a72d","Type":"ContainerStarted","Data":"db44dcf838c16afb440c15cbe17dda3f4102ff8b4647c950d56ae53f14dd1b27"} Nov 03 15:40:50.741836 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:50.741616 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7319-77b693" event={"ID":"8647e935-6cbe-4c09-abad-60e2b2a6a72d","Type":"ContainerStarted","Data":"0592f0c633d4388b4457d68466c8a9e4adfc90b69de275ed904d087a770b2355"} Nov 03 15:40:50.865097 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:50.865030 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7319-77b693" podStartSLOduration=1.865013184 podStartE2EDuration="1.865013184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:40:50.859381072 +0000 UTC m=+1958.290338877" watchObservedRunningTime="2026-11-03 15:40:50.865013184 +0000 UTC m=+1958.295970993" Nov 03 15:40:51.761256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:51.760132 4290 generic.go:356] "Generic (PLEG): container finished" podID="e8ba51c9-602b-47a6-bca9-5f9fa7547276" containerID="6fc19d5d046165da0348bda2470167353008907a464f6b85d4979119b67c8db3" exitCode=0 Nov 03 15:40:51.761256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:51.761030 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" event={"ID":"e8ba51c9-602b-47a6-bca9-5f9fa7547276","Type":"ContainerDied","Data":"6fc19d5d046165da0348bda2470167353008907a464f6b85d4979119b67c8db3"} Nov 03 15:40:51.837896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:51.837872 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" Nov 03 15:40:51.948855 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:51.948827 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8ba51c9-602b-47a6-bca9-5f9fa7547276-kube-api-access-mmqbc\" (UniqueName: \"kubernetes.io/projected/e8ba51c9-602b-47a6-bca9-5f9fa7547276-kube-api-access-mmqbc\") pod \"e8ba51c9-602b-47a6-bca9-5f9fa7547276\" (UID: \"e8ba51c9-602b-47a6-bca9-5f9fa7547276\") " Nov 03 15:40:51.949129 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:51.949092 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8ba51c9-602b-47a6-bca9-5f9fa7547276-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ba51c9-602b-47a6-bca9-5f9fa7547276-plugins-config-volume\") pod \"e8ba51c9-602b-47a6-bca9-5f9fa7547276\" (UID: \"e8ba51c9-602b-47a6-bca9-5f9fa7547276\") " Nov 03 15:40:51.949543 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:51.949514 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ba51c9-602b-47a6-bca9-5f9fa7547276-plugins-config-volume" pod "e8ba51c9-602b-47a6-bca9-5f9fa7547276" (UID: "e8ba51c9-602b-47a6-bca9-5f9fa7547276"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:40:51.949933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:51.949909 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ba51c9-602b-47a6-bca9-5f9fa7547276-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:51.951716 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:51.951674 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ba51c9-602b-47a6-bca9-5f9fa7547276-kube-api-access-mmqbc" pod "e8ba51c9-602b-47a6-bca9-5f9fa7547276" (UID: "e8ba51c9-602b-47a6-bca9-5f9fa7547276"). InnerVolumeSpecName "kube-api-access-mmqbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:40:52.008617 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.008592 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-1b0910" Nov 03 15:40:52.051501 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.051418 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmqbc\" (UniqueName: \"kubernetes.io/projected/e8ba51c9-602b-47a6-bca9-5f9fa7547276-kube-api-access-mmqbc\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:52.153104 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.153074 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996ff5c7-5d26-4fae-8b03-34c8ef551ff9-kube-api-access-nrpfs\" (UniqueName: \"kubernetes.io/projected/996ff5c7-5d26-4fae-8b03-34c8ef551ff9-kube-api-access-nrpfs\") pod \"996ff5c7-5d26-4fae-8b03-34c8ef551ff9\" (UID: \"996ff5c7-5d26-4fae-8b03-34c8ef551ff9\") " Nov 03 15:40:52.155366 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.155333 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996ff5c7-5d26-4fae-8b03-34c8ef551ff9-kube-api-access-nrpfs" pod "996ff5c7-5d26-4fae-8b03-34c8ef551ff9" (UID: "996ff5c7-5d26-4fae-8b03-34c8ef551ff9"). InnerVolumeSpecName "kube-api-access-nrpfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:40:52.255593 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.254828 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrpfs\" (UniqueName: \"kubernetes.io/projected/996ff5c7-5d26-4fae-8b03-34c8ef551ff9-kube-api-access-nrpfs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:52.434508 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.434472 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7319-77b693"] Nov 03 15:40:52.443006 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.442970 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7319-77b693"] Nov 03 15:40:52.680930 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.680895 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="996ff5c7-5d26-4fae-8b03-34c8ef551ff9" path="/var/lib/kubelet/pods/996ff5c7-5d26-4fae-8b03-34c8ef551ff9/volumes" Nov 03 15:40:52.771779 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.771752 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" Nov 03 15:40:52.772257 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.771799 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl" event={"ID":"e8ba51c9-602b-47a6-bca9-5f9fa7547276","Type":"ContainerDied","Data":"015a1452feb5481ea737b55af4d04a5ce39f2af1a6ecaf81057f2bb54139fc14"} Nov 03 15:40:52.772257 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.771872 4290 scope.go:122] "RemoveContainer" containerID="6fc19d5d046165da0348bda2470167353008907a464f6b85d4979119b67c8db3" Nov 03 15:40:52.773871 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.773842 4290 generic.go:356] "Generic (PLEG): container finished" podID="996ff5c7-5d26-4fae-8b03-34c8ef551ff9" containerID="8c1013e951e8c09902b510cff12695d7e1daec439feee4e7d0a93d79598f27c6" exitCode=137 Nov 03 15:40:52.773994 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.773910 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-1b0910" Nov 03 15:40:52.774399 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.774340 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7319-77b693" podUID="8647e935-6cbe-4c09-abad-60e2b2a6a72d" containerName="test-curl-7319-77b693" containerID="cri-o://db44dcf838c16afb440c15cbe17dda3f4102ff8b4647c950d56ae53f14dd1b27" gracePeriod=2 Nov 03 15:40:52.785430 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.785409 4290 scope.go:122] "RemoveContainer" containerID="8c1013e951e8c09902b510cff12695d7e1daec439feee4e7d0a93d79598f27c6" Nov 03 15:40:52.795306 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.795275 4290 scope.go:122] "RemoveContainer" containerID="8c1013e951e8c09902b510cff12695d7e1daec439feee4e7d0a93d79598f27c6" Nov 03 15:40:52.795559 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:52.795542 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1013e951e8c09902b510cff12695d7e1daec439feee4e7d0a93d79598f27c6\": container with ID starting with 8c1013e951e8c09902b510cff12695d7e1daec439feee4e7d0a93d79598f27c6 not found: ID does not exist" containerID="8c1013e951e8c09902b510cff12695d7e1daec439feee4e7d0a93d79598f27c6" Nov 03 15:40:52.795608 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.795566 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1013e951e8c09902b510cff12695d7e1daec439feee4e7d0a93d79598f27c6"} err="failed to get container status \"8c1013e951e8c09902b510cff12695d7e1daec439feee4e7d0a93d79598f27c6\": rpc error: code = NotFound desc = could not find container \"8c1013e951e8c09902b510cff12695d7e1daec439feee4e7d0a93d79598f27c6\": container with ID starting with 8c1013e951e8c09902b510cff12695d7e1daec439feee4e7d0a93d79598f27c6 not found: ID does not exist" Nov 03 15:40:52.944120 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.944036 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl"] Nov 03 15:40:52.951547 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:52.951522 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-dxgzl"] Nov 03 15:40:54.683585 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:54.683543 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8ba51c9-602b-47a6-bca9-5f9fa7547276" path="/var/lib/kubelet/pods/e8ba51c9-602b-47a6-bca9-5f9fa7547276/volumes" Nov 03 15:40:55.023898 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:55.023873 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-77b693" Nov 03 15:40:55.111546 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:55.111513 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8647e935-6cbe-4c09-abad-60e2b2a6a72d-kube-api-access-hjx7l\" (UniqueName: \"kubernetes.io/projected/8647e935-6cbe-4c09-abad-60e2b2a6a72d-kube-api-access-hjx7l\") pod \"8647e935-6cbe-4c09-abad-60e2b2a6a72d\" (UID: \"8647e935-6cbe-4c09-abad-60e2b2a6a72d\") " Nov 03 15:40:55.113843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:55.113805 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8647e935-6cbe-4c09-abad-60e2b2a6a72d-kube-api-access-hjx7l" pod "8647e935-6cbe-4c09-abad-60e2b2a6a72d" (UID: "8647e935-6cbe-4c09-abad-60e2b2a6a72d"). InnerVolumeSpecName "kube-api-access-hjx7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:40:55.213196 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:55.213158 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjx7l\" (UniqueName: \"kubernetes.io/projected/8647e935-6cbe-4c09-abad-60e2b2a6a72d-kube-api-access-hjx7l\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:40:55.812422 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:55.812382 4290 generic.go:356] "Generic (PLEG): container finished" podID="8647e935-6cbe-4c09-abad-60e2b2a6a72d" containerID="db44dcf838c16afb440c15cbe17dda3f4102ff8b4647c950d56ae53f14dd1b27" exitCode=137 Nov 03 15:40:55.812422 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:55.812427 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-77b693" Nov 03 15:40:55.812929 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:55.812478 4290 scope.go:122] "RemoveContainer" containerID="db44dcf838c16afb440c15cbe17dda3f4102ff8b4647c950d56ae53f14dd1b27" Nov 03 15:40:55.823685 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:55.823665 4290 scope.go:122] "RemoveContainer" containerID="db44dcf838c16afb440c15cbe17dda3f4102ff8b4647c950d56ae53f14dd1b27" Nov 03 15:40:55.823966 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:55.823948 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db44dcf838c16afb440c15cbe17dda3f4102ff8b4647c950d56ae53f14dd1b27\": container with ID starting with db44dcf838c16afb440c15cbe17dda3f4102ff8b4647c950d56ae53f14dd1b27 not found: ID does not exist" containerID="db44dcf838c16afb440c15cbe17dda3f4102ff8b4647c950d56ae53f14dd1b27" Nov 03 15:40:55.824008 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:55.823973 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db44dcf838c16afb440c15cbe17dda3f4102ff8b4647c950d56ae53f14dd1b27"} err="failed to get container status \"db44dcf838c16afb440c15cbe17dda3f4102ff8b4647c950d56ae53f14dd1b27\": rpc error: code = NotFound desc = could not find container \"db44dcf838c16afb440c15cbe17dda3f4102ff8b4647c950d56ae53f14dd1b27\": container with ID starting with db44dcf838c16afb440c15cbe17dda3f4102ff8b4647c950d56ae53f14dd1b27 not found: ID does not exist" Nov 03 15:40:55.871897 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:40:55.871859 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-77b693\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="8647e935-6cbe-4c09-abad-60e2b2a6a72d" pod="opendatahub/test-curl-7319-77b693" Nov 03 15:40:56.680283 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:40:56.680252 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8647e935-6cbe-4c09-abad-60e2b2a6a72d" path="/var/lib/kubelet/pods/8647e935-6cbe-4c09-abad-60e2b2a6a72d/volumes" Nov 03 15:41:01.947862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:01.947829 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g"] Nov 03 15:41:01.948568 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:01.948554 4290 cpu_manager.go:414] "containerMap: removing container" podUID="8647e935-6cbe-4c09-abad-60e2b2a6a72d" containerName="test-curl-7319-77b693" Nov 03 15:41:01.948627 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:01.948569 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8647e935-6cbe-4c09-abad-60e2b2a6a72d" containerName="test-curl-7319-77b693" Nov 03 15:41:01.948627 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:01.948588 4290 cpu_manager.go:414] "containerMap: removing container" podUID="e8ba51c9-602b-47a6-bca9-5f9fa7547276" containerName="payload-processing" Nov 03 15:41:01.948627 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:01.948593 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ba51c9-602b-47a6-bca9-5f9fa7547276" containerName="payload-processing" Nov 03 15:41:01.948752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:01.948727 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="8647e935-6cbe-4c09-abad-60e2b2a6a72d" containerName="test-curl-7319-77b693" Nov 03 15:41:01.948752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:01.948738 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ba51c9-602b-47a6-bca9-5f9fa7547276" containerName="payload-processing" Nov 03 15:41:01.953675 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:01.953657 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" Nov 03 15:41:02.007275 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:02.007244 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:41:02.012849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:02.012818 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g"] Nov 03 15:41:02.056520 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:02.056478 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:41:02.115509 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:02.115474 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9wt6\" (UniqueName: \"kubernetes.io/projected/d61fa98e-f543-4c48-91ad-9abccd424a9d-kube-api-access-r9wt6\") pod \"maas-api-revoke-sub-a0dd9a75-8hc4g\" (UID: \"d61fa98e-f543-4c48-91ad-9abccd424a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" Nov 03 15:41:02.115731 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:02.115644 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61fa98e-f543-4c48-91ad-9abccd424a9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0dd9a75-8hc4g\" (UID: \"d61fa98e-f543-4c48-91ad-9abccd424a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" Nov 03 15:41:02.115854 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:02.115836 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61fa98e-f543-4c48-91ad-9abccd424a9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0dd9a75-8hc4g\" (UID: \"d61fa98e-f543-4c48-91ad-9abccd424a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" Nov 03 15:41:02.217521 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:02.217445 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61fa98e-f543-4c48-91ad-9abccd424a9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0dd9a75-8hc4g\" (UID: \"d61fa98e-f543-4c48-91ad-9abccd424a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" Nov 03 15:41:02.217819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:02.217591 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9wt6\" (UniqueName: \"kubernetes.io/projected/d61fa98e-f543-4c48-91ad-9abccd424a9d-kube-api-access-r9wt6\") pod \"maas-api-revoke-sub-a0dd9a75-8hc4g\" (UID: \"d61fa98e-f543-4c48-91ad-9abccd424a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" Nov 03 15:41:02.217819 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:02.217694 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61fa98e-f543-4c48-91ad-9abccd424a9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0dd9a75-8hc4g\" (UID: \"d61fa98e-f543-4c48-91ad-9abccd424a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" Nov 03 15:41:02.218333 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:02.218309 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61fa98e-f543-4c48-91ad-9abccd424a9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0dd9a75-8hc4g\" (UID: \"d61fa98e-f543-4c48-91ad-9abccd424a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" Nov 03 15:41:02.219902 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:02.219883 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61fa98e-f543-4c48-91ad-9abccd424a9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0dd9a75-8hc4g\" (UID: \"d61fa98e-f543-4c48-91ad-9abccd424a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" Nov 03 15:41:02.279718 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:02.279669 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9wt6\" (UniqueName: \"kubernetes.io/projected/d61fa98e-f543-4c48-91ad-9abccd424a9d-kube-api-access-r9wt6\") pod \"maas-api-revoke-sub-a0dd9a75-8hc4g\" (UID: \"d61fa98e-f543-4c48-91ad-9abccd424a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" Nov 03 15:41:02.569113 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:02.569039 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" Nov 03 15:41:02.958459 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:41:02.958431 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61fa98e_f543_4c48_91ad_9abccd424a9d.slice/crio-5666dd55c451f2874be679ae0ed7ca5758fdd0e2c64b618bec2983eebf5a5a5b WatchSource:0}: Error finding container 5666dd55c451f2874be679ae0ed7ca5758fdd0e2c64b618bec2983eebf5a5a5b: Status 404 returned error can't find the container with id 5666dd55c451f2874be679ae0ed7ca5758fdd0e2c64b618bec2983eebf5a5a5b Nov 03 15:41:02.960878 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:02.960853 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g"] Nov 03 15:41:03.902916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:03.902881 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" event={"ID":"d61fa98e-f543-4c48-91ad-9abccd424a9d","Type":"ContainerStarted","Data":"67d567777658282fdb3a7922197bae35ad2eb48cc0b288a9cf67fc8f2a3b787d"} Nov 03 15:41:03.903058 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:03.902944 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" event={"ID":"d61fa98e-f543-4c48-91ad-9abccd424a9d","Type":"ContainerStarted","Data":"5666dd55c451f2874be679ae0ed7ca5758fdd0e2c64b618bec2983eebf5a5a5b"} Nov 03 15:41:04.025973 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:04.025910 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" podStartSLOduration=3.025891173 podStartE2EDuration="3.025891173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:41:04.025599748 +0000 UTC m=+1971.456557560" watchObservedRunningTime="2026-11-03 15:41:04.025891173 +0000 UTC m=+1971.456848981" Nov 03 15:41:04.912382 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:04.912339 4290 generic.go:356] "Generic (PLEG): container finished" podID="d61fa98e-f543-4c48-91ad-9abccd424a9d" containerID="67d567777658282fdb3a7922197bae35ad2eb48cc0b288a9cf67fc8f2a3b787d" exitCode=0 Nov 03 15:41:04.912568 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:04.912398 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" event={"ID":"d61fa98e-f543-4c48-91ad-9abccd424a9d","Type":"ContainerDied","Data":"67d567777658282fdb3a7922197bae35ad2eb48cc0b288a9cf67fc8f2a3b787d"} Nov 03 15:41:06.096336 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:06.096310 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" Nov 03 15:41:06.175914 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:06.175844 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d61fa98e-f543-4c48-91ad-9abccd424a9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61fa98e-f543-4c48-91ad-9abccd424a9d-maas-api-xks-ca\") pod \"d61fa98e-f543-4c48-91ad-9abccd424a9d\" (UID: \"d61fa98e-f543-4c48-91ad-9abccd424a9d\") " Nov 03 15:41:06.176089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:06.175975 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61fa98e-f543-4c48-91ad-9abccd424a9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61fa98e-f543-4c48-91ad-9abccd424a9d-maas-api-openshift-ca\") pod \"d61fa98e-f543-4c48-91ad-9abccd424a9d\" (UID: \"d61fa98e-f543-4c48-91ad-9abccd424a9d\") " Nov 03 15:41:06.176089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:06.176034 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61fa98e-f543-4c48-91ad-9abccd424a9d-kube-api-access-r9wt6\" (UniqueName: \"kubernetes.io/projected/d61fa98e-f543-4c48-91ad-9abccd424a9d-kube-api-access-r9wt6\") pod \"d61fa98e-f543-4c48-91ad-9abccd424a9d\" (UID: \"d61fa98e-f543-4c48-91ad-9abccd424a9d\") " Nov 03 15:41:06.176358 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:06.176324 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61fa98e-f543-4c48-91ad-9abccd424a9d-maas-api-openshift-ca" pod "d61fa98e-f543-4c48-91ad-9abccd424a9d" (UID: "d61fa98e-f543-4c48-91ad-9abccd424a9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:41:06.176565 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:06.176549 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61fa98e-f543-4c48-91ad-9abccd424a9d-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:06.178217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:06.178192 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d61fa98e-f543-4c48-91ad-9abccd424a9d-maas-api-xks-ca" pod "d61fa98e-f543-4c48-91ad-9abccd424a9d" (UID: "d61fa98e-f543-4c48-91ad-9abccd424a9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:41:06.178319 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:06.178235 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61fa98e-f543-4c48-91ad-9abccd424a9d-kube-api-access-r9wt6" pod "d61fa98e-f543-4c48-91ad-9abccd424a9d" (UID: "d61fa98e-f543-4c48-91ad-9abccd424a9d"). InnerVolumeSpecName "kube-api-access-r9wt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:41:06.278486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:06.278451 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61fa98e-f543-4c48-91ad-9abccd424a9d-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:06.278486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:06.278481 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9wt6\" (UniqueName: \"kubernetes.io/projected/d61fa98e-f543-4c48-91ad-9abccd424a9d-kube-api-access-r9wt6\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:06.931047 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:06.931010 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" Nov 03 15:41:06.931047 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:06.931037 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g" event={"ID":"d61fa98e-f543-4c48-91ad-9abccd424a9d","Type":"ContainerDied","Data":"5666dd55c451f2874be679ae0ed7ca5758fdd0e2c64b618bec2983eebf5a5a5b"} Nov 03 15:41:06.931253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:06.931091 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5666dd55c451f2874be679ae0ed7ca5758fdd0e2c64b618bec2983eebf5a5a5b" Nov 03 15:41:08.706991 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:41:08.706898 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:41:17.297016 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.296974 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42"] Nov 03 15:41:17.298156 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.298132 4290 cpu_manager.go:414] "containerMap: removing container" podUID="d61fa98e-f543-4c48-91ad-9abccd424a9d" containerName="revoke-keys" Nov 03 15:41:17.298230 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.298160 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61fa98e-f543-4c48-91ad-9abccd424a9d" containerName="revoke-keys" Nov 03 15:41:17.298424 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.298410 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61fa98e-f543-4c48-91ad-9abccd424a9d" containerName="revoke-keys" Nov 03 15:41:17.304400 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.304378 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" Nov 03 15:41:17.360100 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.360052 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42"] Nov 03 15:41:17.401450 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.401414 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:41:17.401664 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.401458 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:41:17.456321 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.456245 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c79915bb-9rz42\" (UID: \"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" Nov 03 15:41:17.456473 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.456369 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbpvr\" (UniqueName: \"kubernetes.io/projected/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-kube-api-access-wbpvr\") pod \"maas-api-revoke-sub-c79915bb-9rz42\" (UID: \"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" Nov 03 15:41:17.456598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.456577 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c79915bb-9rz42\" (UID: \"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" Nov 03 15:41:17.558523 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.558483 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbpvr\" (UniqueName: \"kubernetes.io/projected/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-kube-api-access-wbpvr\") pod \"maas-api-revoke-sub-c79915bb-9rz42\" (UID: \"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" Nov 03 15:41:17.558716 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.558683 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c79915bb-9rz42\" (UID: \"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" Nov 03 15:41:17.558844 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.558828 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c79915bb-9rz42\" (UID: \"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" Nov 03 15:41:17.559358 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.559330 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c79915bb-9rz42\" (UID: \"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" Nov 03 15:41:17.561367 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.561342 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c79915bb-9rz42\" (UID: \"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" Nov 03 15:41:17.620395 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.620354 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbpvr\" (UniqueName: \"kubernetes.io/projected/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-kube-api-access-wbpvr\") pod \"maas-api-revoke-sub-c79915bb-9rz42\" (UID: \"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" Nov 03 15:41:17.622056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:17.622034 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" Nov 03 15:41:18.033927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:18.033833 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42"] Nov 03 15:41:18.041490 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:41:18.041456 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d04a5ca_1479_4cb9_992d_a296d7ecbd2d.slice/crio-77e3b1520517e1defdfefe4dc3bd917414503c4e15e5e6fe54ce967866feda8d WatchSource:0}: Error finding container 77e3b1520517e1defdfefe4dc3bd917414503c4e15e5e6fe54ce967866feda8d: Status 404 returned error can't find the container with id 77e3b1520517e1defdfefe4dc3bd917414503c4e15e5e6fe54ce967866feda8d Nov 03 15:41:19.047430 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:19.047397 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" event={"ID":"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d","Type":"ContainerStarted","Data":"4ea9f0a5b010122891bfdbf8144cc663018c5f4a72a86d0e9a6b8afaf6816f62"} Nov 03 15:41:19.047924 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:19.047451 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" event={"ID":"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d","Type":"ContainerStarted","Data":"77e3b1520517e1defdfefe4dc3bd917414503c4e15e5e6fe54ce967866feda8d"} Nov 03 15:41:19.234077 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:19.209353 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" podStartSLOduration=2.209332563 podStartE2EDuration="2.209332563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:41:19.165191098 +0000 UTC m=+1986.596148905" watchObservedRunningTime="2026-11-03 15:41:19.209332563 +0000 UTC m=+1986.640290372" Nov 03 15:41:20.060727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:20.060470 4290 generic.go:356] "Generic (PLEG): container finished" podID="3d04a5ca-1479-4cb9-992d-a296d7ecbd2d" containerID="4ea9f0a5b010122891bfdbf8144cc663018c5f4a72a86d0e9a6b8afaf6816f62" exitCode=0 Nov 03 15:41:20.060727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:20.060509 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" event={"ID":"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d","Type":"ContainerDied","Data":"4ea9f0a5b010122891bfdbf8144cc663018c5f4a72a86d0e9a6b8afaf6816f62"} Nov 03 15:41:21.853406 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:21.853325 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp"] Nov 03 15:41:21.854165 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:21.854133 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" containerID="cri-o://de177b1f7f97a7c6bbb3c271eb4ca9bd44ef4011d62b408e72a46dde532e284d" gracePeriod=30 Nov 03 15:41:21.881690 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:21.881638 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q"] Nov 03 15:41:21.891910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:21.891867 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" podUID="5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" containerName="istio-proxy" containerID="cri-o://970553885c9dbec874bdf27dbf8f8707faec4c819e96b33205da59251f352671" gracePeriod=30 Nov 03 15:41:22.032722 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:22.031894 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" Nov 03 15:41:22.101990 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:22.101958 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-maas-api-openshift-ca\") pod \"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d\" (UID: \"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d\") " Nov 03 15:41:22.102184 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:22.102134 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-maas-api-xks-ca\") pod \"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d\" (UID: \"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d\") " Nov 03 15:41:22.102256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:22.102215 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-kube-api-access-wbpvr\" (UniqueName: \"kubernetes.io/projected/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-kube-api-access-wbpvr\") pod \"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d\" (UID: \"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d\") " Nov 03 15:41:22.102393 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:22.102367 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-maas-api-openshift-ca" pod "3d04a5ca-1479-4cb9-992d-a296d7ecbd2d" (UID: "3d04a5ca-1479-4cb9-992d-a296d7ecbd2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:41:22.102837 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:22.102809 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:22.105768 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:22.104692 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-maas-api-xks-ca" pod "3d04a5ca-1479-4cb9-992d-a296d7ecbd2d" (UID: "3d04a5ca-1479-4cb9-992d-a296d7ecbd2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:41:22.105768 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:22.104912 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-kube-api-access-wbpvr" pod "3d04a5ca-1479-4cb9-992d-a296d7ecbd2d" (UID: "3d04a5ca-1479-4cb9-992d-a296d7ecbd2d"). InnerVolumeSpecName "kube-api-access-wbpvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:41:22.131777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:22.131741 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" event={"ID":"3d04a5ca-1479-4cb9-992d-a296d7ecbd2d","Type":"ContainerDied","Data":"77e3b1520517e1defdfefe4dc3bd917414503c4e15e5e6fe54ce967866feda8d"} Nov 03 15:41:22.131915 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:22.131809 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77e3b1520517e1defdfefe4dc3bd917414503c4e15e5e6fe54ce967866feda8d" Nov 03 15:41:22.131915 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:22.131757 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42" Nov 03 15:41:22.204974 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:22.204940 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:22.204974 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:22.204970 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbpvr\" (UniqueName: \"kubernetes.io/projected/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d-kube-api-access-wbpvr\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:22.328390 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:22.328351 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx"] Nov 03 15:41:22.328642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:22.328618 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" podUID="12553958-5457-4c66-b011-431e954fef89" containerName="istio-proxy" containerID="cri-o://223b7fb759f6e329c0c7b408e1f0caa8a809146082363a517919c81d8ba1dade" gracePeriod=30 Nov 03 15:41:23.112773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:23.112108 4290 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:41:23.112773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:23.112171 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" podUID="12553958-5457-4c66-b011-431e954fef89" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:41:24.728458 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:24.728409 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p"] Nov 03 15:41:24.779337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:24.779304 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-7b3fd3-4vl7p"] Nov 03 15:41:26.680847 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:26.680813 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="928cb260-2bf9-4c3e-b72d-cf22412df2ff" path="/var/lib/kubelet/pods/928cb260-2bf9-4c3e-b72d-cf22412df2ff/volumes" Nov 03 15:41:27.186086 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.186010 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" event={"ID":"5d888abb-c170-4809-a0d3-642d20b77628","Type":"ContainerDied","Data":"de177b1f7f97a7c6bbb3c271eb4ca9bd44ef4011d62b408e72a46dde532e284d"} Nov 03 15:41:27.186283 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.186263 4290 generic.go:356] "Generic (PLEG): container finished" podID="5d888abb-c170-4809-a0d3-642d20b77628" containerID="de177b1f7f97a7c6bbb3c271eb4ca9bd44ef4011d62b408e72a46dde532e284d" exitCode=0 Nov 03 15:41:27.188609 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.188580 4290 generic.go:356] "Generic (PLEG): container finished" podID="5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" containerID="970553885c9dbec874bdf27dbf8f8707faec4c819e96b33205da59251f352671" exitCode=0 Nov 03 15:41:27.188907 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.188658 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" event={"ID":"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9","Type":"ContainerDied","Data":"970553885c9dbec874bdf27dbf8f8707faec4c819e96b33205da59251f352671"} Nov 03 15:41:27.239898 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.239872 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:41:27.244894 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.244866 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:41:27.385393 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.385361 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istiod-ca-cert\") pod \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " Nov 03 15:41:27.385393 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.385397 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d888abb-c170-4809-a0d3-642d20b77628-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d888abb-c170-4809-a0d3-642d20b77628-istio-podinfo\") pod \"5d888abb-c170-4809-a0d3-642d20b77628\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " Nov 03 15:41:27.385557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.385494 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-workload-certs\") pod \"5d888abb-c170-4809-a0d3-642d20b77628\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " Nov 03 15:41:27.385557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.385540 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-credential-socket\") pod \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " Nov 03 15:41:27.385677 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.385588 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d888abb-c170-4809-a0d3-642d20b77628-istio-token\" (UniqueName: \"kubernetes.io/projected/5d888abb-c170-4809-a0d3-642d20b77628-istio-token\") pod \"5d888abb-c170-4809-a0d3-642d20b77628\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " Nov 03 15:41:27.385677 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.385636 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-token\" (UniqueName: \"kubernetes.io/projected/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-token\") pod \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " Nov 03 15:41:27.385847 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.385721 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-workload-socket\") pod \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " Nov 03 15:41:27.385847 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.385786 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-podinfo\") pod \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " Nov 03 15:41:27.385847 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.385817 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-workload-certs\") pod \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " Nov 03 15:41:27.385847 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.385830 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-workload-certs" pod "5d888abb-c170-4809-a0d3-642d20b77628" (UID: "5d888abb-c170-4809-a0d3-642d20b77628"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:27.386314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.385882 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-credential-socket" pod "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" (UID: "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:27.386314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.385896 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d888abb-c170-4809-a0d3-642d20b77628-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d888abb-c170-4809-a0d3-642d20b77628-istiod-ca-cert\") pod \"5d888abb-c170-4809-a0d3-642d20b77628\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " Nov 03 15:41:27.386314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.385899 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istiod-ca-cert" pod "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" (UID: "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:41:27.386314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.385943 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-envoy\") pod \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " Nov 03 15:41:27.386314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.385975 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-workload-socket\") pod \"5d888abb-c170-4809-a0d3-642d20b77628\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " Nov 03 15:41:27.386314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.386014 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-istio-envoy\") pod \"5d888abb-c170-4809-a0d3-642d20b77628\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " Nov 03 15:41:27.386314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.386042 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-kube-api-access-hdmq9\" (UniqueName: \"kubernetes.io/projected/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-kube-api-access-hdmq9\") pod \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " Nov 03 15:41:27.386314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.386093 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-data\") pod \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\" (UID: \"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9\") " Nov 03 15:41:27.386314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.386159 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-credential-socket\") pod \"5d888abb-c170-4809-a0d3-642d20b77628\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " Nov 03 15:41:27.386314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.386012 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-workload-socket" pod "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" (UID: "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:27.386314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.386157 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-workload-certs" pod "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" (UID: "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:27.386314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.386192 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d888abb-c170-4809-a0d3-642d20b77628-kube-api-access-kzxph\" (UniqueName: \"kubernetes.io/projected/5d888abb-c170-4809-a0d3-642d20b77628-kube-api-access-kzxph\") pod \"5d888abb-c170-4809-a0d3-642d20b77628\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " Nov 03 15:41:27.386314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.386258 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-istio-data\") pod \"5d888abb-c170-4809-a0d3-642d20b77628\" (UID: \"5d888abb-c170-4809-a0d3-642d20b77628\") " Nov 03 15:41:27.387363 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.386335 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-workload-socket" pod "5d888abb-c170-4809-a0d3-642d20b77628" (UID: "5d888abb-c170-4809-a0d3-642d20b77628"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:27.387363 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.387008 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-data" pod "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" (UID: "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:27.387363 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.387244 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-credential-socket" pod "5d888abb-c170-4809-a0d3-642d20b77628" (UID: "5d888abb-c170-4809-a0d3-642d20b77628"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:27.387527 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.387455 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-istio-data" pod "5d888abb-c170-4809-a0d3-642d20b77628" (UID: "5d888abb-c170-4809-a0d3-642d20b77628"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:27.387587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.387555 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d888abb-c170-4809-a0d3-642d20b77628-istiod-ca-cert" pod "5d888abb-c170-4809-a0d3-642d20b77628" (UID: "5d888abb-c170-4809-a0d3-642d20b77628"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:41:27.388355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.387858 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.388355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.388019 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.388355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.388062 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.388355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.388075 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.388355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.388089 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d888abb-c170-4809-a0d3-642d20b77628-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.388355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.388105 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.388355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.388118 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.388355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.388131 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.388355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.388145 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.388355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.388159 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.389149 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.389113 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-envoy" pod "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" (UID: "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:27.389217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.389175 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-podinfo" pod "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" (UID: "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:41:27.390790 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.390669 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d888abb-c170-4809-a0d3-642d20b77628-istio-podinfo" pod "5d888abb-c170-4809-a0d3-642d20b77628" (UID: "5d888abb-c170-4809-a0d3-642d20b77628"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:41:27.390907 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.390827 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d888abb-c170-4809-a0d3-642d20b77628-istio-token" pod "5d888abb-c170-4809-a0d3-642d20b77628" (UID: "5d888abb-c170-4809-a0d3-642d20b77628"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:41:27.390907 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.390860 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-token" pod "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" (UID: "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:41:27.390907 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.390873 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-istio-envoy" pod "5d888abb-c170-4809-a0d3-642d20b77628" (UID: "5d888abb-c170-4809-a0d3-642d20b77628"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:27.391044 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.390943 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-kube-api-access-hdmq9" pod "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" (UID: "5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9"). InnerVolumeSpecName "kube-api-access-hdmq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:41:27.391998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.391972 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d888abb-c170-4809-a0d3-642d20b77628-kube-api-access-kzxph" pod "5d888abb-c170-4809-a0d3-642d20b77628" (UID: "5d888abb-c170-4809-a0d3-642d20b77628"). InnerVolumeSpecName "kube-api-access-kzxph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:41:27.490140 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.490056 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d888abb-c170-4809-a0d3-642d20b77628-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.490140 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.490087 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdmq9\" (UniqueName: \"kubernetes.io/projected/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-kube-api-access-hdmq9\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.490140 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.490098 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzxph\" (UniqueName: \"kubernetes.io/projected/5d888abb-c170-4809-a0d3-642d20b77628-kube-api-access-kzxph\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.490140 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.490107 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d888abb-c170-4809-a0d3-642d20b77628-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.490140 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.490117 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d888abb-c170-4809-a0d3-642d20b77628-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.490140 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.490126 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.490140 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.490134 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.490140 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.490142 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.616192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.616168 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:41:27.792925 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.792845 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12553958-5457-4c66-b011-431e954fef89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12553958-5457-4c66-b011-431e954fef89-istio-podinfo\") pod \"12553958-5457-4c66-b011-431e954fef89\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " Nov 03 15:41:27.792925 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.792913 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-workload-socket\") pod \"12553958-5457-4c66-b011-431e954fef89\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " Nov 03 15:41:27.793445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.792944 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-istio-data\") pod \"12553958-5457-4c66-b011-431e954fef89\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " Nov 03 15:41:27.793445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.792995 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12553958-5457-4c66-b011-431e954fef89-kube-api-access-j8jmg\" (UniqueName: \"kubernetes.io/projected/12553958-5457-4c66-b011-431e954fef89-kube-api-access-j8jmg\") pod \"12553958-5457-4c66-b011-431e954fef89\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " Nov 03 15:41:27.793445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.793110 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12553958-5457-4c66-b011-431e954fef89-istio-token\" (UniqueName: \"kubernetes.io/projected/12553958-5457-4c66-b011-431e954fef89-istio-token\") pod \"12553958-5457-4c66-b011-431e954fef89\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " Nov 03 15:41:27.793445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.793151 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-workload-certs\") pod \"12553958-5457-4c66-b011-431e954fef89\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " Nov 03 15:41:27.793445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.793217 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12553958-5457-4c66-b011-431e954fef89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12553958-5457-4c66-b011-431e954fef89-istiod-ca-cert\") pod \"12553958-5457-4c66-b011-431e954fef89\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " Nov 03 15:41:27.793445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.793234 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-workload-socket" pod "12553958-5457-4c66-b011-431e954fef89" (UID: "12553958-5457-4c66-b011-431e954fef89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:27.793445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.793260 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-istio-envoy\") pod \"12553958-5457-4c66-b011-431e954fef89\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " Nov 03 15:41:27.793445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.793282 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-istio-data" pod "12553958-5457-4c66-b011-431e954fef89" (UID: "12553958-5457-4c66-b011-431e954fef89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:27.793445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.793355 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-credential-socket\") pod \"12553958-5457-4c66-b011-431e954fef89\" (UID: \"12553958-5457-4c66-b011-431e954fef89\") " Nov 03 15:41:27.794232 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.793631 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-workload-certs" pod "12553958-5457-4c66-b011-431e954fef89" (UID: "12553958-5457-4c66-b011-431e954fef89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:27.794232 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.793660 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-credential-socket" pod "12553958-5457-4c66-b011-431e954fef89" (UID: "12553958-5457-4c66-b011-431e954fef89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:27.794232 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.793944 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12553958-5457-4c66-b011-431e954fef89-istiod-ca-cert" pod "12553958-5457-4c66-b011-431e954fef89" (UID: "12553958-5457-4c66-b011-431e954fef89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:41:27.794653 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.794625 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.794653 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.794648 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.794922 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.794662 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.794922 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.794674 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12553958-5457-4c66-b011-431e954fef89-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.794922 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.794687 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.795903 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.795881 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12553958-5457-4c66-b011-431e954fef89-istio-podinfo" pod "12553958-5457-4c66-b011-431e954fef89" (UID: "12553958-5457-4c66-b011-431e954fef89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:41:27.795964 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.795913 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12553958-5457-4c66-b011-431e954fef89-istio-token" pod "12553958-5457-4c66-b011-431e954fef89" (UID: "12553958-5457-4c66-b011-431e954fef89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:41:27.796047 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.796028 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-istio-envoy" pod "12553958-5457-4c66-b011-431e954fef89" (UID: "12553958-5457-4c66-b011-431e954fef89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:27.796115 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.796102 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12553958-5457-4c66-b011-431e954fef89-kube-api-access-j8jmg" pod "12553958-5457-4c66-b011-431e954fef89" (UID: "12553958-5457-4c66-b011-431e954fef89"). InnerVolumeSpecName "kube-api-access-j8jmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:41:27.896116 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.896033 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12553958-5457-4c66-b011-431e954fef89-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.896116 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.896059 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8jmg\" (UniqueName: \"kubernetes.io/projected/12553958-5457-4c66-b011-431e954fef89-kube-api-access-j8jmg\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.896116 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.896070 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12553958-5457-4c66-b011-431e954fef89-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:27.896116 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:27.896078 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12553958-5457-4c66-b011-431e954fef89-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:28.199516 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.199483 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" Nov 03 15:41:28.199516 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.199509 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp" event={"ID":"5d888abb-c170-4809-a0d3-642d20b77628","Type":"ContainerDied","Data":"59137be573767137a481cba73252bf694430dc09c12cd7f3ceb55eb855bb0ff5"} Nov 03 15:41:28.199830 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.199573 4290 scope.go:122] "RemoveContainer" containerID="de177b1f7f97a7c6bbb3c271eb4ca9bd44ef4011d62b408e72a46dde532e284d" Nov 03 15:41:28.201468 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.201446 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" Nov 03 15:41:28.201593 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.201462 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q" event={"ID":"5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9","Type":"ContainerDied","Data":"526f1352329e5c03ec9ed6a747c5237b5df0a0a6e338c59e16c38f0843bc9613"} Nov 03 15:41:28.203231 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.203200 4290 generic.go:356] "Generic (PLEG): container finished" podID="12553958-5457-4c66-b011-431e954fef89" containerID="223b7fb759f6e329c0c7b408e1f0caa8a809146082363a517919c81d8ba1dade" exitCode=0 Nov 03 15:41:28.203372 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.203317 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" event={"ID":"12553958-5457-4c66-b011-431e954fef89","Type":"ContainerDied","Data":"223b7fb759f6e329c0c7b408e1f0caa8a809146082363a517919c81d8ba1dade"} Nov 03 15:41:28.203372 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.203358 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" event={"ID":"12553958-5457-4c66-b011-431e954fef89","Type":"ContainerDied","Data":"bfe5db684b12bafe8e7edde6c8859cc94597e9ee7071627938b36928c87e8678"} Nov 03 15:41:28.203518 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.203398 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx" Nov 03 15:41:28.215261 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.214847 4290 scope.go:122] "RemoveContainer" containerID="970553885c9dbec874bdf27dbf8f8707faec4c819e96b33205da59251f352671" Nov 03 15:41:28.227165 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.227138 4290 scope.go:122] "RemoveContainer" containerID="223b7fb759f6e329c0c7b408e1f0caa8a809146082363a517919c81d8ba1dade" Nov 03 15:41:28.238734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.238714 4290 scope.go:122] "RemoveContainer" containerID="223b7fb759f6e329c0c7b408e1f0caa8a809146082363a517919c81d8ba1dade" Nov 03 15:41:28.239056 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:41:28.239031 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"223b7fb759f6e329c0c7b408e1f0caa8a809146082363a517919c81d8ba1dade\": container with ID starting with 223b7fb759f6e329c0c7b408e1f0caa8a809146082363a517919c81d8ba1dade not found: ID does not exist" containerID="223b7fb759f6e329c0c7b408e1f0caa8a809146082363a517919c81d8ba1dade" Nov 03 15:41:28.239129 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.239066 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223b7fb759f6e329c0c7b408e1f0caa8a809146082363a517919c81d8ba1dade"} err="failed to get container status \"223b7fb759f6e329c0c7b408e1f0caa8a809146082363a517919c81d8ba1dade\": rpc error: code = NotFound desc = could not find container \"223b7fb759f6e329c0c7b408e1f0caa8a809146082363a517919c81d8ba1dade\": container with ID starting with 223b7fb759f6e329c0c7b408e1f0caa8a809146082363a517919c81d8ba1dade not found: ID does not exist" Nov 03 15:41:28.368207 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.368177 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp"] Nov 03 15:41:28.418165 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.418129 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-bcqxp"] Nov 03 15:41:28.484345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.484261 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx"] Nov 03 15:41:28.489802 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.489775 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-nrsnx"] Nov 03 15:41:28.601048 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.601015 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q"] Nov 03 15:41:28.605887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.605859 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-zkh4q"] Nov 03 15:41:28.680598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.680565 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12553958-5457-4c66-b011-431e954fef89" path="/var/lib/kubelet/pods/12553958-5457-4c66-b011-431e954fef89/volumes" Nov 03 15:41:28.681148 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.681131 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" path="/var/lib/kubelet/pods/5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9/volumes" Nov 03 15:41:28.681617 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:28.681601 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d888abb-c170-4809-a0d3-642d20b77628" path="/var/lib/kubelet/pods/5d888abb-c170-4809-a0d3-642d20b77628/volumes" Nov 03 15:41:33.255338 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.255300 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t"] Nov 03 15:41:33.256405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.256382 4290 cpu_manager.go:414] "containerMap: removing container" podUID="5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" containerName="istio-proxy" Nov 03 15:41:33.256405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.256403 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" containerName="istio-proxy" Nov 03 15:41:33.256630 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.256447 4290 cpu_manager.go:414] "containerMap: removing container" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" Nov 03 15:41:33.256630 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.256455 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" Nov 03 15:41:33.256630 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.256470 4290 cpu_manager.go:414] "containerMap: removing container" podUID="3d04a5ca-1479-4cb9-992d-a296d7ecbd2d" containerName="revoke-keys" Nov 03 15:41:33.256630 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.256478 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d04a5ca-1479-4cb9-992d-a296d7ecbd2d" containerName="revoke-keys" Nov 03 15:41:33.256630 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.256494 4290 cpu_manager.go:414] "containerMap: removing container" podUID="12553958-5457-4c66-b011-431e954fef89" containerName="istio-proxy" Nov 03 15:41:33.256630 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.256503 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12553958-5457-4c66-b011-431e954fef89" containerName="istio-proxy" Nov 03 15:41:33.256984 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.256642 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a18e3ac-9d85-4ac3-a1dd-310fbbe263e9" containerName="istio-proxy" Nov 03 15:41:33.256984 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.256658 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d888abb-c170-4809-a0d3-642d20b77628" containerName="istio-proxy" Nov 03 15:41:33.256984 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.256669 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d04a5ca-1479-4cb9-992d-a296d7ecbd2d" containerName="revoke-keys" Nov 03 15:41:33.256984 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.256683 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="12553958-5457-4c66-b011-431e954fef89" containerName="istio-proxy" Nov 03 15:41:33.262332 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.262301 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" Nov 03 15:41:33.315845 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.315813 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t"] Nov 03 15:41:33.359260 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.359227 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:41:33.359497 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.359280 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:41:33.370094 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.370059 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/419f8bf7-bc55-41f2-b38d-662f1d834a5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t\" (UID: \"419f8bf7-bc55-41f2-b38d-662f1d834a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" Nov 03 15:41:33.370270 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.370121 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/419f8bf7-bc55-41f2-b38d-662f1d834a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t\" (UID: \"419f8bf7-bc55-41f2-b38d-662f1d834a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" Nov 03 15:41:33.370359 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.370334 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l77wr\" (UniqueName: \"kubernetes.io/projected/419f8bf7-bc55-41f2-b38d-662f1d834a5c-kube-api-access-l77wr\") pod \"maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t\" (UID: \"419f8bf7-bc55-41f2-b38d-662f1d834a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" Nov 03 15:41:33.471854 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.471816 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/419f8bf7-bc55-41f2-b38d-662f1d834a5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t\" (UID: \"419f8bf7-bc55-41f2-b38d-662f1d834a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" Nov 03 15:41:33.472047 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.471868 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/419f8bf7-bc55-41f2-b38d-662f1d834a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t\" (UID: \"419f8bf7-bc55-41f2-b38d-662f1d834a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" Nov 03 15:41:33.472047 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.472002 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l77wr\" (UniqueName: \"kubernetes.io/projected/419f8bf7-bc55-41f2-b38d-662f1d834a5c-kube-api-access-l77wr\") pod \"maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t\" (UID: \"419f8bf7-bc55-41f2-b38d-662f1d834a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" Nov 03 15:41:33.472637 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.472550 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/419f8bf7-bc55-41f2-b38d-662f1d834a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t\" (UID: \"419f8bf7-bc55-41f2-b38d-662f1d834a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" Nov 03 15:41:33.474560 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.474536 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/419f8bf7-bc55-41f2-b38d-662f1d834a5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t\" (UID: \"419f8bf7-bc55-41f2-b38d-662f1d834a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" Nov 03 15:41:33.534367 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.534304 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l77wr\" (UniqueName: \"kubernetes.io/projected/419f8bf7-bc55-41f2-b38d-662f1d834a5c-kube-api-access-l77wr\") pod \"maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t\" (UID: \"419f8bf7-bc55-41f2-b38d-662f1d834a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" Nov 03 15:41:33.577987 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.577960 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" Nov 03 15:41:33.979682 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:33.979517 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t"] Nov 03 15:41:34.261295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:34.261260 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" event={"ID":"419f8bf7-bc55-41f2-b38d-662f1d834a5c","Type":"ContainerStarted","Data":"9bef9cc581d04f0daecfe819893e75f6e5ee4efdc1860e94a41bead7f1f91e1e"} Nov 03 15:41:35.272243 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:35.272144 4290 generic.go:356] "Generic (PLEG): container finished" podID="419f8bf7-bc55-41f2-b38d-662f1d834a5c" containerID="2030d386f300888cbece7fed695c7da98d17add9bb2f8cefa30241a74eaff9b2" exitCode=0 Nov 03 15:41:35.272243 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:35.272228 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" event={"ID":"419f8bf7-bc55-41f2-b38d-662f1d834a5c","Type":"ContainerDied","Data":"2030d386f300888cbece7fed695c7da98d17add9bb2f8cefa30241a74eaff9b2"} Nov 03 15:41:36.450313 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:36.450288 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" Nov 03 15:41:36.514448 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:36.514406 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/419f8bf7-bc55-41f2-b38d-662f1d834a5c-kube-api-access-l77wr\" (UniqueName: \"kubernetes.io/projected/419f8bf7-bc55-41f2-b38d-662f1d834a5c-kube-api-access-l77wr\") pod \"419f8bf7-bc55-41f2-b38d-662f1d834a5c\" (UID: \"419f8bf7-bc55-41f2-b38d-662f1d834a5c\") " Nov 03 15:41:36.514632 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:36.514553 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/419f8bf7-bc55-41f2-b38d-662f1d834a5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/419f8bf7-bc55-41f2-b38d-662f1d834a5c-maas-api-openshift-ca\") pod \"419f8bf7-bc55-41f2-b38d-662f1d834a5c\" (UID: \"419f8bf7-bc55-41f2-b38d-662f1d834a5c\") " Nov 03 15:41:36.514786 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:36.514766 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/419f8bf7-bc55-41f2-b38d-662f1d834a5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/419f8bf7-bc55-41f2-b38d-662f1d834a5c-maas-api-xks-ca\") pod \"419f8bf7-bc55-41f2-b38d-662f1d834a5c\" (UID: \"419f8bf7-bc55-41f2-b38d-662f1d834a5c\") " Nov 03 15:41:36.514994 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:36.514960 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/419f8bf7-bc55-41f2-b38d-662f1d834a5c-maas-api-openshift-ca" pod "419f8bf7-bc55-41f2-b38d-662f1d834a5c" (UID: "419f8bf7-bc55-41f2-b38d-662f1d834a5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:41:36.515357 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:36.515339 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/419f8bf7-bc55-41f2-b38d-662f1d834a5c-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:36.516863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:36.516840 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419f8bf7-bc55-41f2-b38d-662f1d834a5c-kube-api-access-l77wr" pod "419f8bf7-bc55-41f2-b38d-662f1d834a5c" (UID: "419f8bf7-bc55-41f2-b38d-662f1d834a5c"). InnerVolumeSpecName "kube-api-access-l77wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:41:36.516924 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:36.516871 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/419f8bf7-bc55-41f2-b38d-662f1d834a5c-maas-api-xks-ca" pod "419f8bf7-bc55-41f2-b38d-662f1d834a5c" (UID: "419f8bf7-bc55-41f2-b38d-662f1d834a5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:41:36.616268 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:36.616181 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/419f8bf7-bc55-41f2-b38d-662f1d834a5c-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:36.616268 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:36.616210 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l77wr\" (UniqueName: \"kubernetes.io/projected/419f8bf7-bc55-41f2-b38d-662f1d834a5c-kube-api-access-l77wr\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:37.302008 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:37.301977 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" Nov 03 15:41:37.302008 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:37.301988 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t" event={"ID":"419f8bf7-bc55-41f2-b38d-662f1d834a5c","Type":"ContainerDied","Data":"9bef9cc581d04f0daecfe819893e75f6e5ee4efdc1860e94a41bead7f1f91e1e"} Nov 03 15:41:37.302244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:37.302079 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bef9cc581d04f0daecfe819893e75f6e5ee4efdc1860e94a41bead7f1f91e1e" Nov 03 15:41:37.701720 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:37.701661 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw"] Nov 03 15:41:37.707590 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:37.707562 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv"] Nov 03 15:41:38.704950 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:41:38.704908 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:41:45.296985 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:45.296947 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4"] Nov 03 15:41:45.408995 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:45.408952 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm"] Nov 03 15:41:50.297422 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:50.297373 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" podUID="b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8" containerName="payload-processing" containerID="cri-o://a1aa01937825caba9d90bdf28fdcbe8ba4f808e90ba344e077e6d7f4378a5f3d" gracePeriod=25 Nov 03 15:41:50.410508 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:50.409769 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" podUID="247dd294-ca8a-49ca-a75e-a2d65ceb7991" containerName="payload-pre-processing" containerID="cri-o://6938df5dbfcdf3a0eebe0897ab060eabcb2e09fac158eb4ce8c522ad4594e8b4" gracePeriod=25 Nov 03 15:41:52.740725 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:52.737323 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" containerID="cri-o://7e0787a54e7ca4982f707483fa104e8255d9795e1085d65b135176d793197269" gracePeriod=45 Nov 03 15:41:52.819386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:52.819055 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" containerID="cri-o://ab85753f61ff2270f809e1dd859bb2adb9d1c9333c89414787e283eb57a89b12" gracePeriod=45 Nov 03 15:41:52.999752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:52.999299 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-2sbg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:41:52.999752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:52.999377 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" podUID="871baa2b-806e-4033-a553-277c509a3c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:41:53.218747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.217207 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:41:53.242081 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.242049 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:41:53.310989 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.310898 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-dshm\") pod \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " Nov 03 15:41:53.310989 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.310952 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-model-cache\") pod \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " Nov 03 15:41:53.310989 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.310988 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2de980d4-51ff-4fad-a85a-2f45af90a354-tls-certs\" (UniqueName: \"kubernetes.io/secret/2de980d4-51ff-4fad-a85a-2f45af90a354-tls-certs\") pod \"2de980d4-51ff-4fad-a85a-2f45af90a354\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " Nov 03 15:41:53.311276 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.311064 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9d9187-4d88-4a6b-bd10-c29febff43bd-kube-api-access-gpwm2\" (UniqueName: \"kubernetes.io/projected/2b9d9187-4d88-4a6b-bd10-c29febff43bd-kube-api-access-gpwm2\") pod \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " Nov 03 15:41:53.311276 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.311165 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-tmp-dir\") pod \"2de980d4-51ff-4fad-a85a-2f45af90a354\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " Nov 03 15:41:53.311276 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.311203 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de980d4-51ff-4fad-a85a-2f45af90a354-kube-api-access-gm7m6\" (UniqueName: \"kubernetes.io/projected/2de980d4-51ff-4fad-a85a-2f45af90a354-kube-api-access-gm7m6\") pod \"2de980d4-51ff-4fad-a85a-2f45af90a354\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " Nov 03 15:41:53.311458 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.311327 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-home\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-home\") pod \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " Nov 03 15:41:53.311458 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.311369 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-model-cache\") pod \"2de980d4-51ff-4fad-a85a-2f45af90a354\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " Nov 03 15:41:53.311458 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.311409 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-dshm\") pod \"2de980d4-51ff-4fad-a85a-2f45af90a354\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " Nov 03 15:41:53.311610 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.311506 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9d9187-4d88-4a6b-bd10-c29febff43bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9d9187-4d88-4a6b-bd10-c29febff43bd-tls-certs\") pod \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " Nov 03 15:41:53.311610 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.311537 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-tmp-dir\") pod \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\" (UID: \"2b9d9187-4d88-4a6b-bd10-c29febff43bd\") " Nov 03 15:41:53.311610 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.311580 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-home\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-home\") pod \"2de980d4-51ff-4fad-a85a-2f45af90a354\" (UID: \"2de980d4-51ff-4fad-a85a-2f45af90a354\") " Nov 03 15:41:53.313377 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.313222 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-tmp-dir" pod "2b9d9187-4d88-4a6b-bd10-c29febff43bd" (UID: "2b9d9187-4d88-4a6b-bd10-c29febff43bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.314446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.313394 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-home" pod "2de980d4-51ff-4fad-a85a-2f45af90a354" (UID: "2de980d4-51ff-4fad-a85a-2f45af90a354"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.314446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.313613 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-tmp-dir" pod "2de980d4-51ff-4fad-a85a-2f45af90a354" (UID: "2de980d4-51ff-4fad-a85a-2f45af90a354"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.314446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.313905 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-home" pod "2b9d9187-4d88-4a6b-bd10-c29febff43bd" (UID: "2b9d9187-4d88-4a6b-bd10-c29febff43bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.314446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.313917 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-model-cache" pod "2b9d9187-4d88-4a6b-bd10-c29febff43bd" (UID: "2b9d9187-4d88-4a6b-bd10-c29febff43bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.314446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.314214 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-model-cache" pod "2de980d4-51ff-4fad-a85a-2f45af90a354" (UID: "2de980d4-51ff-4fad-a85a-2f45af90a354"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.316495 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.316399 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2de980d4-51ff-4fad-a85a-2f45af90a354-tls-certs" pod "2de980d4-51ff-4fad-a85a-2f45af90a354" (UID: "2de980d4-51ff-4fad-a85a-2f45af90a354"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:41:53.316495 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.316437 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9d9187-4d88-4a6b-bd10-c29febff43bd-tls-certs" pod "2b9d9187-4d88-4a6b-bd10-c29febff43bd" (UID: "2b9d9187-4d88-4a6b-bd10-c29febff43bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:41:53.317896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.317860 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-dshm" pod "2b9d9187-4d88-4a6b-bd10-c29febff43bd" (UID: "2b9d9187-4d88-4a6b-bd10-c29febff43bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.318155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.318110 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9d9187-4d88-4a6b-bd10-c29febff43bd-kube-api-access-gpwm2" pod "2b9d9187-4d88-4a6b-bd10-c29febff43bd" (UID: "2b9d9187-4d88-4a6b-bd10-c29febff43bd"). InnerVolumeSpecName "kube-api-access-gpwm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:41:53.318571 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.318539 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de980d4-51ff-4fad-a85a-2f45af90a354-kube-api-access-gm7m6" pod "2de980d4-51ff-4fad-a85a-2f45af90a354" (UID: "2de980d4-51ff-4fad-a85a-2f45af90a354"). InnerVolumeSpecName "kube-api-access-gm7m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:41:53.320345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.320318 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-dshm" pod "2de980d4-51ff-4fad-a85a-2f45af90a354" (UID: "2de980d4-51ff-4fad-a85a-2f45af90a354"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.414462 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.414420 4290 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9d9187-4d88-4a6b-bd10-c29febff43bd-tls-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.414462 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.414466 4290 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-tmp-dir\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.415244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.414481 4290 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-home\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.415244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.414493 4290 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-dshm\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.415244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.414507 4290 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-model-cache\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.415244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.414524 4290 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2de980d4-51ff-4fad-a85a-2f45af90a354-tls-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.415244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.414539 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpwm2\" (UniqueName: \"kubernetes.io/projected/2b9d9187-4d88-4a6b-bd10-c29febff43bd-kube-api-access-gpwm2\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.415244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.414554 4290 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-tmp-dir\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.415244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.414569 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm7m6\" (UniqueName: \"kubernetes.io/projected/2de980d4-51ff-4fad-a85a-2f45af90a354-kube-api-access-gm7m6\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.415244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.414582 4290 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9d9187-4d88-4a6b-bd10-c29febff43bd-home\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.415244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.414596 4290 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-model-cache\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.415244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.414609 4290 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2de980d4-51ff-4fad-a85a-2f45af90a354-dshm\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.644386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.644345 4290 generic.go:356] "Generic (PLEG): container finished" podID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerID="ab85753f61ff2270f809e1dd859bb2adb9d1c9333c89414787e283eb57a89b12" exitCode=0 Nov 03 15:41:53.644551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.644402 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" Nov 03 15:41:53.644551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.644428 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" event={"ID":"2b9d9187-4d88-4a6b-bd10-c29febff43bd","Type":"ContainerDied","Data":"ab85753f61ff2270f809e1dd859bb2adb9d1c9333c89414787e283eb57a89b12"} Nov 03 15:41:53.644551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.644479 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw" event={"ID":"2b9d9187-4d88-4a6b-bd10-c29febff43bd","Type":"ContainerDied","Data":"83103dcdb885b2b1ea2053e0fa9069159172f210be3e30468aefd38874a28b16"} Nov 03 15:41:53.644551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.644528 4290 scope.go:122] "RemoveContainer" containerID="ab85753f61ff2270f809e1dd859bb2adb9d1c9333c89414787e283eb57a89b12" Nov 03 15:41:53.646413 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.646252 4290 generic.go:356] "Generic (PLEG): container finished" podID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerID="7e0787a54e7ca4982f707483fa104e8255d9795e1085d65b135176d793197269" exitCode=0 Nov 03 15:41:53.646413 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.646292 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" event={"ID":"2de980d4-51ff-4fad-a85a-2f45af90a354","Type":"ContainerDied","Data":"7e0787a54e7ca4982f707483fa104e8255d9795e1085d65b135176d793197269"} Nov 03 15:41:53.646413 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.646307 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" Nov 03 15:41:53.646413 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.646326 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv" event={"ID":"2de980d4-51ff-4fad-a85a-2f45af90a354","Type":"ContainerDied","Data":"b09da9f3769f66793b32ccfe5d921e71af44392715e24d3968401e21f89ef1bd"} Nov 03 15:41:53.657479 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.657460 4290 scope.go:122] "RemoveContainer" containerID="ab85753f61ff2270f809e1dd859bb2adb9d1c9333c89414787e283eb57a89b12" Nov 03 15:41:53.657797 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:41:53.657778 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab85753f61ff2270f809e1dd859bb2adb9d1c9333c89414787e283eb57a89b12\": container with ID starting with ab85753f61ff2270f809e1dd859bb2adb9d1c9333c89414787e283eb57a89b12 not found: ID does not exist" containerID="ab85753f61ff2270f809e1dd859bb2adb9d1c9333c89414787e283eb57a89b12" Nov 03 15:41:53.657863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.657805 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab85753f61ff2270f809e1dd859bb2adb9d1c9333c89414787e283eb57a89b12"} err="failed to get container status \"ab85753f61ff2270f809e1dd859bb2adb9d1c9333c89414787e283eb57a89b12\": rpc error: code = NotFound desc = could not find container \"ab85753f61ff2270f809e1dd859bb2adb9d1c9333c89414787e283eb57a89b12\": container with ID starting with ab85753f61ff2270f809e1dd859bb2adb9d1c9333c89414787e283eb57a89b12 not found: ID does not exist" Nov 03 15:41:53.657863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.657827 4290 scope.go:122] "RemoveContainer" containerID="7e0787a54e7ca4982f707483fa104e8255d9795e1085d65b135176d793197269" Nov 03 15:41:53.669225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.669206 4290 scope.go:122] "RemoveContainer" containerID="7e0787a54e7ca4982f707483fa104e8255d9795e1085d65b135176d793197269" Nov 03 15:41:53.669580 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:41:53.669557 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0787a54e7ca4982f707483fa104e8255d9795e1085d65b135176d793197269\": container with ID starting with 7e0787a54e7ca4982f707483fa104e8255d9795e1085d65b135176d793197269 not found: ID does not exist" containerID="7e0787a54e7ca4982f707483fa104e8255d9795e1085d65b135176d793197269" Nov 03 15:41:53.669688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.669586 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0787a54e7ca4982f707483fa104e8255d9795e1085d65b135176d793197269"} err="failed to get container status \"7e0787a54e7ca4982f707483fa104e8255d9795e1085d65b135176d793197269\": rpc error: code = NotFound desc = could not find container \"7e0787a54e7ca4982f707483fa104e8255d9795e1085d65b135176d793197269\": container with ID starting with 7e0787a54e7ca4982f707483fa104e8255d9795e1085d65b135176d793197269 not found: ID does not exist" Nov 03 15:41:53.818302 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.818263 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw"] Nov 03 15:41:53.822934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.822904 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/distinct-2-w5-34e8c5-kserve-5c8cbc9bbf-nfgdw"] Nov 03 15:41:53.940743 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.940616 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv"] Nov 03 15:41:53.945581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:53.945554 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/distinct-w5-34e8c5-kserve-5bd9679b98-sq8cv"] Nov 03 15:41:54.681587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:54.681541 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" path="/var/lib/kubelet/pods/2b9d9187-4d88-4a6b-bd10-c29febff43bd/volumes" Nov 03 15:41:54.682177 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:54.682153 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" path="/var/lib/kubelet/pods/2de980d4-51ff-4fad-a85a-2f45af90a354/volumes" Nov 03 15:41:56.225312 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:56.225275 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" podUID="247dd294-ca8a-49ca-a75e-a2d65ceb7991" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:41:57.231924 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:41:57.231888 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" podUID="b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:42:00.552895 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.552854 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc"] Nov 03 15:42:00.554008 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.553982 4290 cpu_manager.go:414] "containerMap: removing container" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" Nov 03 15:42:00.554008 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.554008 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" Nov 03 15:42:00.554216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.554076 4290 cpu_manager.go:414] "containerMap: removing container" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" Nov 03 15:42:00.554216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.554085 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" Nov 03 15:42:00.554216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.554126 4290 cpu_manager.go:414] "containerMap: removing container" podUID="419f8bf7-bc55-41f2-b38d-662f1d834a5c" containerName="revoke-keys" Nov 03 15:42:00.554216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.554134 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419f8bf7-bc55-41f2-b38d-662f1d834a5c" containerName="revoke-keys" Nov 03 15:42:00.554415 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.554316 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de980d4-51ff-4fad-a85a-2f45af90a354" containerName="main" Nov 03 15:42:00.554415 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.554335 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="419f8bf7-bc55-41f2-b38d-662f1d834a5c" containerName="revoke-keys" Nov 03 15:42:00.554415 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.554351 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9d9187-4d88-4a6b-bd10-c29febff43bd" containerName="main" Nov 03 15:42:00.559653 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.559630 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.611425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.611390 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h"] Nov 03 15:42:00.616298 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.616269 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.618546 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.618514 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a99bae-0003-49ad-99e7-6a53b9d80867-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.618691 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.618560 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.618691 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.618639 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.618843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.618745 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.618843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.618805 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.618951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.618863 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.618951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.618909 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqk2d\" (UniqueName: \"kubernetes.io/projected/44a99bae-0003-49ad-99e7-6a53b9d80867-kube-api-access-rqk2d\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.619065 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.618984 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.619120 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.619062 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.631189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.631164 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-90fcca-openshift-default-dockercfg-g7lcc\"" Nov 03 15:42:00.653757 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.653717 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc"] Nov 03 15:42:00.701577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.701545 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h"] Nov 03 15:42:00.721091 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721045 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.721297 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721111 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.721297 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721219 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.721297 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721285 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.721483 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721332 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a99bae-0003-49ad-99e7-6a53b9d80867-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.721483 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721371 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.721483 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721389 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.721483 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721421 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.721732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721528 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.721732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721543 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.721732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721561 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.721732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721588 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.721732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721591 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.721732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721694 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.722068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721830 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.722068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721874 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.722068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721887 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.722068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.721963 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frw5h\" (UniqueName: \"kubernetes.io/projected/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-kube-api-access-frw5h\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.722068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.722006 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.722068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.722049 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqk2d\" (UniqueName: \"kubernetes.io/projected/44a99bae-0003-49ad-99e7-6a53b9d80867-kube-api-access-rqk2d\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.722377 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.722073 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.722377 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.722172 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a99bae-0003-49ad-99e7-6a53b9d80867-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.723506 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.722467 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.726203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.726177 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.726369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.726353 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.785124 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.785087 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqk2d\" (UniqueName: \"kubernetes.io/projected/44a99bae-0003-49ad-99e7-6a53b9d80867-kube-api-access-rqk2d\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.785274 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.785244 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.823552 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.823447 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frw5h\" (UniqueName: \"kubernetes.io/projected/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-kube-api-access-frw5h\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.823552 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.823524 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.823824 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.823587 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.823824 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.823677 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.823824 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.823756 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.823824 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.823785 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.823824 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.823816 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.824068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.823854 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.824068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.823879 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.824068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.824023 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.824270 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.824239 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.824339 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.824267 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.824419 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.824399 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.824951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.824928 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.826311 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.826288 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.826396 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.826381 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.877711 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.877657 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:00.888256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.888222 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.888528 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.888504 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frw5h\" (UniqueName: \"kubernetes.io/projected/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-kube-api-access-frw5h\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-97d6h\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:00.932633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:00.932599 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:01.329557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.329497 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc"] Nov 03 15:42:01.329690 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:42:01.329657 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a99bae_0003_49ad_99e7_6a53b9d80867.slice/crio-032f134ec341ae5ee0dd4556e2bb9e25ae33c602b9b2aa87153740b4582599e9 WatchSource:0}: Error finding container 032f134ec341ae5ee0dd4556e2bb9e25ae33c602b9b2aa87153740b4582599e9: Status 404 returned error can't find the container with id 032f134ec341ae5ee0dd4556e2bb9e25ae33c602b9b2aa87153740b4582599e9 Nov 03 15:42:01.332141 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.332107 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:42:01.332233 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.332173 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:42:01.332233 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.332203 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:42:01.442853 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.442817 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h"] Nov 03 15:42:01.446048 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.446004 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:42:01.446159 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.446097 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:42:01.446159 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.446141 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:42:01.729002 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.728889 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" event={"ID":"44a99bae-0003-49ad-99e7-6a53b9d80867","Type":"ContainerStarted","Data":"5a1af2e6c69280e43f3ee0183c58a3b4c9b037e4eb67884b959a87b7b332b3a1"} Nov 03 15:42:01.729002 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.728965 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" event={"ID":"44a99bae-0003-49ad-99e7-6a53b9d80867","Type":"ContainerStarted","Data":"032f134ec341ae5ee0dd4556e2bb9e25ae33c602b9b2aa87153740b4582599e9"} Nov 03 15:42:01.731093 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.731063 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" event={"ID":"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0","Type":"ContainerStarted","Data":"171cb99bc27c60c11a56062dfb0f22637a4c563de812d0c374733e841c667337"} Nov 03 15:42:01.731225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.731116 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" event={"ID":"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0","Type":"ContainerStarted","Data":"4a3f6525d07ef9cb37d6f60dae758a81e86392afe02e2ae0a06d011eb52ad5e7"} Nov 03 15:42:01.858429 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.858374 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" podStartSLOduration=1.858359632 podStartE2EDuration="1.858359632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:42:01.851967311 +0000 UTC m=+2029.282925118" watchObservedRunningTime="2026-11-03 15:42:01.858359632 +0000 UTC m=+2029.289317438" Nov 03 15:42:01.878365 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.878330 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:01.933164 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.933133 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:01.982161 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:01.982043 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" podStartSLOduration=1.982025203 podStartE2EDuration="1.982025203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:42:01.978072746 +0000 UTC m=+2029.409030554" watchObservedRunningTime="2026-11-03 15:42:01.982025203 +0000 UTC m=+2029.412983008" Nov 03 15:42:02.897843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:02.897788 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 15:42:02.898363 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:02.897870 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" podUID="44a99bae-0003-49ad-99e7-6a53b9d80867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 15:42:02.936919 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:02.936882 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 15:42:02.937210 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:02.937176 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" podUID="b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 15:42:03.881141 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:03.881100 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 15:42:03.881522 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:03.881489 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" podUID="44a99bae-0003-49ad-99e7-6a53b9d80867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 15:42:03.933385 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:03.933352 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 15:42:03.933943 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:03.933910 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" podUID="b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 15:42:04.889729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:04.886473 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 15:42:04.889729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:04.886534 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" podUID="44a99bae-0003-49ad-99e7-6a53b9d80867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 15:42:04.933481 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:04.933433 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 15:42:04.933943 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:04.933500 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" podUID="b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 15:42:05.905063 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:05.905024 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 15:42:05.905352 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:05.905322 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" podUID="44a99bae-0003-49ad-99e7-6a53b9d80867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 15:42:05.933623 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:05.933592 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 15:42:05.934088 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:05.933818 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" podUID="b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 15:42:06.226008 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:06.225918 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" podUID="247dd294-ca8a-49ca-a75e-a2d65ceb7991" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:42:06.878143 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:06.878103 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 15:42:06.878476 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:06.878444 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" podUID="44a99bae-0003-49ad-99e7-6a53b9d80867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 15:42:06.933511 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:06.933477 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 15:42:06.933800 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:06.933776 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" podUID="b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 15:42:07.232967 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:07.232590 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" podUID="b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:42:07.883814 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:07.883766 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:07.884195 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:07.884170 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:07.885031 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:07.885005 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:42:07.938322 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:07.938290 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:07.938874 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:07.938728 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:07.939544 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:07.939522 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:42:08.704817 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:42:08.704714 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:42:10.637301 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.637255 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" Nov 03 15:42:10.681739 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.681597 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8-plugins-config-volume\") pod \"b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8\" (UID: \"b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8\") " Nov 03 15:42:10.681739 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.681737 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8-kube-api-access-jb4fb\" (UniqueName: \"kubernetes.io/projected/b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8-kube-api-access-jb4fb\") pod \"b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8\" (UID: \"b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8\") " Nov 03 15:42:10.683250 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.683170 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8-plugins-config-volume" pod "b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8" (UID: "b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:42:10.685157 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.685133 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8-kube-api-access-jb4fb" pod "b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8" (UID: "b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8"). InnerVolumeSpecName "kube-api-access-jb4fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:42:10.706435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.706409 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" Nov 03 15:42:10.783969 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.783937 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247dd294-ca8a-49ca-a75e-a2d65ceb7991-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247dd294-ca8a-49ca-a75e-a2d65ceb7991-plugins-config-volume\") pod \"247dd294-ca8a-49ca-a75e-a2d65ceb7991\" (UID: \"247dd294-ca8a-49ca-a75e-a2d65ceb7991\") " Nov 03 15:42:10.784174 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.784040 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247dd294-ca8a-49ca-a75e-a2d65ceb7991-kube-api-access-d27nh\" (UniqueName: \"kubernetes.io/projected/247dd294-ca8a-49ca-a75e-a2d65ceb7991-kube-api-access-d27nh\") pod \"247dd294-ca8a-49ca-a75e-a2d65ceb7991\" (UID: \"247dd294-ca8a-49ca-a75e-a2d65ceb7991\") " Nov 03 15:42:10.784455 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.784418 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247dd294-ca8a-49ca-a75e-a2d65ceb7991-plugins-config-volume" pod "247dd294-ca8a-49ca-a75e-a2d65ceb7991" (UID: "247dd294-ca8a-49ca-a75e-a2d65ceb7991"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:42:10.785592 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.785566 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:10.785716 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.785600 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247dd294-ca8a-49ca-a75e-a2d65ceb7991-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:10.785716 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.785615 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb4fb\" (UniqueName: \"kubernetes.io/projected/b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8-kube-api-access-jb4fb\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:10.786598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.786575 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247dd294-ca8a-49ca-a75e-a2d65ceb7991-kube-api-access-d27nh" pod "247dd294-ca8a-49ca-a75e-a2d65ceb7991" (UID: "247dd294-ca8a-49ca-a75e-a2d65ceb7991"). InnerVolumeSpecName "kube-api-access-d27nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:42:10.865147 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.865060 4290 generic.go:356] "Generic (PLEG): container finished" podID="b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8" containerID="a1aa01937825caba9d90bdf28fdcbe8ba4f808e90ba344e077e6d7f4378a5f3d" exitCode=0 Nov 03 15:42:10.865147 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.865108 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" Nov 03 15:42:10.865335 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.865293 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" event={"ID":"b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8","Type":"ContainerDied","Data":"a1aa01937825caba9d90bdf28fdcbe8ba4f808e90ba344e077e6d7f4378a5f3d"} Nov 03 15:42:10.865372 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.865345 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4" event={"ID":"b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8","Type":"ContainerDied","Data":"4471114e042da66ee3919714c4e97014c1d6e4e33b37b655316e0b13c5d81d43"} Nov 03 15:42:10.865409 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.865374 4290 scope.go:122] "RemoveContainer" containerID="a1aa01937825caba9d90bdf28fdcbe8ba4f808e90ba344e077e6d7f4378a5f3d" Nov 03 15:42:10.867405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.867376 4290 generic.go:356] "Generic (PLEG): container finished" podID="247dd294-ca8a-49ca-a75e-a2d65ceb7991" containerID="6938df5dbfcdf3a0eebe0897ab060eabcb2e09fac158eb4ce8c522ad4594e8b4" exitCode=0 Nov 03 15:42:10.867534 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.867408 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" event={"ID":"247dd294-ca8a-49ca-a75e-a2d65ceb7991","Type":"ContainerDied","Data":"6938df5dbfcdf3a0eebe0897ab060eabcb2e09fac158eb4ce8c522ad4594e8b4"} Nov 03 15:42:10.867534 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.867478 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" Nov 03 15:42:10.867642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.867479 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm" event={"ID":"247dd294-ca8a-49ca-a75e-a2d65ceb7991","Type":"ContainerDied","Data":"c95121354e0fd513132631f2f5cb578bf047574c6c21144c3d8674bd52e4e3ae"} Nov 03 15:42:10.879072 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.879055 4290 scope.go:122] "RemoveContainer" containerID="a1aa01937825caba9d90bdf28fdcbe8ba4f808e90ba344e077e6d7f4378a5f3d" Nov 03 15:42:10.879442 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:42:10.879423 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1aa01937825caba9d90bdf28fdcbe8ba4f808e90ba344e077e6d7f4378a5f3d\": container with ID starting with a1aa01937825caba9d90bdf28fdcbe8ba4f808e90ba344e077e6d7f4378a5f3d not found: ID does not exist" containerID="a1aa01937825caba9d90bdf28fdcbe8ba4f808e90ba344e077e6d7f4378a5f3d" Nov 03 15:42:10.879513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.879453 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1aa01937825caba9d90bdf28fdcbe8ba4f808e90ba344e077e6d7f4378a5f3d"} err="failed to get container status \"a1aa01937825caba9d90bdf28fdcbe8ba4f808e90ba344e077e6d7f4378a5f3d\": rpc error: code = NotFound desc = could not find container \"a1aa01937825caba9d90bdf28fdcbe8ba4f808e90ba344e077e6d7f4378a5f3d\": container with ID starting with a1aa01937825caba9d90bdf28fdcbe8ba4f808e90ba344e077e6d7f4378a5f3d not found: ID does not exist" Nov 03 15:42:10.879513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.879478 4290 scope.go:122] "RemoveContainer" containerID="6938df5dbfcdf3a0eebe0897ab060eabcb2e09fac158eb4ce8c522ad4594e8b4" Nov 03 15:42:10.896521 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.896484 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d27nh\" (UniqueName: \"kubernetes.io/projected/247dd294-ca8a-49ca-a75e-a2d65ceb7991-kube-api-access-d27nh\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:10.900139 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.900105 4290 scope.go:122] "RemoveContainer" containerID="6938df5dbfcdf3a0eebe0897ab060eabcb2e09fac158eb4ce8c522ad4594e8b4" Nov 03 15:42:10.900583 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:42:10.900555 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6938df5dbfcdf3a0eebe0897ab060eabcb2e09fac158eb4ce8c522ad4594e8b4\": container with ID starting with 6938df5dbfcdf3a0eebe0897ab060eabcb2e09fac158eb4ce8c522ad4594e8b4 not found: ID does not exist" containerID="6938df5dbfcdf3a0eebe0897ab060eabcb2e09fac158eb4ce8c522ad4594e8b4" Nov 03 15:42:10.900776 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:10.900738 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6938df5dbfcdf3a0eebe0897ab060eabcb2e09fac158eb4ce8c522ad4594e8b4"} err="failed to get container status \"6938df5dbfcdf3a0eebe0897ab060eabcb2e09fac158eb4ce8c522ad4594e8b4\": rpc error: code = NotFound desc = could not find container \"6938df5dbfcdf3a0eebe0897ab060eabcb2e09fac158eb4ce8c522ad4594e8b4\": container with ID starting with 6938df5dbfcdf3a0eebe0897ab060eabcb2e09fac158eb4ce8c522ad4594e8b4 not found: ID does not exist" Nov 03 15:42:11.012486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:11.012445 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm"] Nov 03 15:42:11.017480 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:11.017452 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-34e8c5-b948f9d8c-jd8bm"] Nov 03 15:42:11.129633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:11.129594 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4"] Nov 03 15:42:11.139695 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:11.139659 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-34e8c5-5f6c7479cc-qk6r4"] Nov 03 15:42:12.680966 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:12.680874 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247dd294-ca8a-49ca-a75e-a2d65ceb7991" path="/var/lib/kubelet/pods/247dd294-ca8a-49ca-a75e-a2d65ceb7991/volumes" Nov 03 15:42:12.681483 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:12.681465 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8" path="/var/lib/kubelet/pods/b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8/volumes" Nov 03 15:42:38.704368 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:42:38.704332 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:42:43.529061 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.529015 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd"] Nov 03 15:42:43.531479 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.531450 4290 cpu_manager.go:414] "containerMap: removing container" podUID="247dd294-ca8a-49ca-a75e-a2d65ceb7991" containerName="payload-pre-processing" Nov 03 15:42:43.531479 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.531481 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247dd294-ca8a-49ca-a75e-a2d65ceb7991" containerName="payload-pre-processing" Nov 03 15:42:43.531719 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.531538 4290 cpu_manager.go:414] "containerMap: removing container" podUID="b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8" containerName="payload-processing" Nov 03 15:42:43.531719 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.531547 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8" containerName="payload-processing" Nov 03 15:42:43.531839 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.531757 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="247dd294-ca8a-49ca-a75e-a2d65ceb7991" containerName="payload-pre-processing" Nov 03 15:42:43.531839 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.531784 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12fd72d-c28d-4fe3-b0d4-c2f97ca1c5c8" containerName="payload-processing" Nov 03 15:42:43.536526 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.536502 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" Nov 03 15:42:43.589292 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.589223 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd"] Nov 03 15:42:43.619066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.619022 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be2add4-64cd-435f-8cc2-69eca189384f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b08a2daf-gwvfd\" (UID: \"8be2add4-64cd-435f-8cc2-69eca189384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" Nov 03 15:42:43.619296 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.619170 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngw6m\" (UniqueName: \"kubernetes.io/projected/8be2add4-64cd-435f-8cc2-69eca189384f-kube-api-access-ngw6m\") pod \"maas-api-revoke-sub-b08a2daf-gwvfd\" (UID: \"8be2add4-64cd-435f-8cc2-69eca189384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" Nov 03 15:42:43.619372 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.619352 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be2add4-64cd-435f-8cc2-69eca189384f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b08a2daf-gwvfd\" (UID: \"8be2add4-64cd-435f-8cc2-69eca189384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" Nov 03 15:42:43.633001 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.632968 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:42:43.633219 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.633169 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:42:43.721165 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.721124 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be2add4-64cd-435f-8cc2-69eca189384f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b08a2daf-gwvfd\" (UID: \"8be2add4-64cd-435f-8cc2-69eca189384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" Nov 03 15:42:43.721380 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.721353 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be2add4-64cd-435f-8cc2-69eca189384f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b08a2daf-gwvfd\" (UID: \"8be2add4-64cd-435f-8cc2-69eca189384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" Nov 03 15:42:43.721451 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.721424 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngw6m\" (UniqueName: \"kubernetes.io/projected/8be2add4-64cd-435f-8cc2-69eca189384f-kube-api-access-ngw6m\") pod \"maas-api-revoke-sub-b08a2daf-gwvfd\" (UID: \"8be2add4-64cd-435f-8cc2-69eca189384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" Nov 03 15:42:43.721945 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.721920 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be2add4-64cd-435f-8cc2-69eca189384f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b08a2daf-gwvfd\" (UID: \"8be2add4-64cd-435f-8cc2-69eca189384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" Nov 03 15:42:43.724385 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.724361 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be2add4-64cd-435f-8cc2-69eca189384f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b08a2daf-gwvfd\" (UID: \"8be2add4-64cd-435f-8cc2-69eca189384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" Nov 03 15:42:43.783791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.783714 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngw6m\" (UniqueName: \"kubernetes.io/projected/8be2add4-64cd-435f-8cc2-69eca189384f-kube-api-access-ngw6m\") pod \"maas-api-revoke-sub-b08a2daf-gwvfd\" (UID: \"8be2add4-64cd-435f-8cc2-69eca189384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" Nov 03 15:42:43.852359 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.852291 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" Nov 03 15:42:43.857014 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.856987 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj"] Nov 03 15:42:43.871171 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:43.871146 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-dtcxj"] Nov 03 15:42:44.244408 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:44.244376 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd"] Nov 03 15:42:44.244950 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:42:44.244926 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be2add4_64cd_435f_8cc2_69eca189384f.slice/crio-a7c6b364e9cc939a64f8f4c99050896d2c52cc1045517258217c0ff36517243d WatchSource:0}: Error finding container a7c6b364e9cc939a64f8f4c99050896d2c52cc1045517258217c0ff36517243d: Status 404 returned error can't find the container with id a7c6b364e9cc939a64f8f4c99050896d2c52cc1045517258217c0ff36517243d Nov 03 15:42:44.680094 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:44.680061 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b9cce47-d408-496c-ab69-58772c005366" path="/var/lib/kubelet/pods/7b9cce47-d408-496c-ab69-58772c005366/volumes" Nov 03 15:42:45.191053 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:45.191015 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" event={"ID":"8be2add4-64cd-435f-8cc2-69eca189384f","Type":"ContainerStarted","Data":"a50f03af0f077fa4ebedfb7b6eead41acd7178bb89127d127ceae747e50f62a6"} Nov 03 15:42:45.191276 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:45.191110 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" event={"ID":"8be2add4-64cd-435f-8cc2-69eca189384f","Type":"ContainerStarted","Data":"a7c6b364e9cc939a64f8f4c99050896d2c52cc1045517258217c0ff36517243d"} Nov 03 15:42:45.317967 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:45.317906 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" podStartSLOduration=2.317888414 podStartE2EDuration="2.317888414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:42:45.312866021 +0000 UTC m=+2072.743823832" watchObservedRunningTime="2026-11-03 15:42:45.317888414 +0000 UTC m=+2072.748846219" Nov 03 15:42:45.998200 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:45.998164 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b"] Nov 03 15:42:46.002673 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.002653 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.056553 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.056512 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.056774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.056567 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n4d8\" (UniqueName: \"kubernetes.io/projected/822b28e8-240d-4cf5-b6de-87900b6c5938-kube-api-access-8n4d8\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.056774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.056653 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.056896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.056802 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.056999 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.056976 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822b28e8-240d-4cf5-b6de-87900b6c5938-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.057051 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.057014 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.057208 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.057185 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.057249 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.057226 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.057374 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.057358 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.159291 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.158924 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.159291 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.159003 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.159291 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.159052 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n4d8\" (UniqueName: \"kubernetes.io/projected/822b28e8-240d-4cf5-b6de-87900b6c5938-kube-api-access-8n4d8\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.159291 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.159081 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.159291 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.159176 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.159291 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.159291 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822b28e8-240d-4cf5-b6de-87900b6c5938-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.160670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.159320 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.160670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.159415 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.160670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.159443 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.160670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.160282 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.160908 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.160717 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822b28e8-240d-4cf5-b6de-87900b6c5938-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.163817 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.163786 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.164136 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.164112 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.164232 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.164159 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.164232 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.164186 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.165740 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.165653 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.203584 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.203549 4290 generic.go:356] "Generic (PLEG): container finished" podID="8be2add4-64cd-435f-8cc2-69eca189384f" containerID="a50f03af0f077fa4ebedfb7b6eead41acd7178bb89127d127ceae747e50f62a6" exitCode=0 Nov 03 15:42:46.203783 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.203639 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" event={"ID":"8be2add4-64cd-435f-8cc2-69eca189384f","Type":"ContainerDied","Data":"a50f03af0f077fa4ebedfb7b6eead41acd7178bb89127d127ceae747e50f62a6"} Nov 03 15:42:46.224181 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.224141 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b"] Nov 03 15:42:46.224959 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.224930 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.225431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.225408 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n4d8\" (UniqueName: \"kubernetes.io/projected/822b28e8-240d-4cf5-b6de-87900b6c5938-kube-api-access-8n4d8\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.318064 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.317973 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:46.447824 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.447789 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m"] Nov 03 15:42:46.455674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.455641 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" Nov 03 15:42:46.509995 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.509955 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m"] Nov 03 15:42:46.566194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.566147 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snzdj\" (UniqueName: \"kubernetes.io/projected/cb83d831-2143-4f54-aa05-6fad48074acf-kube-api-access-snzdj\") pod \"maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m\" (UID: \"cb83d831-2143-4f54-aa05-6fad48074acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" Nov 03 15:42:46.566405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.566379 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb83d831-2143-4f54-aa05-6fad48074acf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m\" (UID: \"cb83d831-2143-4f54-aa05-6fad48074acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" Nov 03 15:42:46.566513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.566494 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb83d831-2143-4f54-aa05-6fad48074acf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m\" (UID: \"cb83d831-2143-4f54-aa05-6fad48074acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" Nov 03 15:42:46.668148 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.668109 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb83d831-2143-4f54-aa05-6fad48074acf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m\" (UID: \"cb83d831-2143-4f54-aa05-6fad48074acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" Nov 03 15:42:46.668376 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.668200 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb83d831-2143-4f54-aa05-6fad48074acf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m\" (UID: \"cb83d831-2143-4f54-aa05-6fad48074acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" Nov 03 15:42:46.668376 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.668338 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snzdj\" (UniqueName: \"kubernetes.io/projected/cb83d831-2143-4f54-aa05-6fad48074acf-kube-api-access-snzdj\") pod \"maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m\" (UID: \"cb83d831-2143-4f54-aa05-6fad48074acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" Nov 03 15:42:46.669172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.669148 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb83d831-2143-4f54-aa05-6fad48074acf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m\" (UID: \"cb83d831-2143-4f54-aa05-6fad48074acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" Nov 03 15:42:46.670878 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.670854 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb83d831-2143-4f54-aa05-6fad48074acf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m\" (UID: \"cb83d831-2143-4f54-aa05-6fad48074acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" Nov 03 15:42:46.738878 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.738842 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snzdj\" (UniqueName: \"kubernetes.io/projected/cb83d831-2143-4f54-aa05-6fad48074acf-kube-api-access-snzdj\") pod \"maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m\" (UID: \"cb83d831-2143-4f54-aa05-6fad48074acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" Nov 03 15:42:46.781842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.781798 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" Nov 03 15:42:46.812793 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:42:46.812761 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822b28e8_240d_4cf5_b6de_87900b6c5938.slice/crio-a202cbe01f9d77810e7db9775287c8f8b568622b27fe356db535732b47d65714 WatchSource:0}: Error finding container a202cbe01f9d77810e7db9775287c8f8b568622b27fe356db535732b47d65714: Status 404 returned error can't find the container with id a202cbe01f9d77810e7db9775287c8f8b568622b27fe356db535732b47d65714 Nov 03 15:42:46.814562 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.814532 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b"] Nov 03 15:42:46.815990 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.815951 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:42:46.816097 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.816025 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:42:46.816097 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:46.816053 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:42:47.188094 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.188059 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m"] Nov 03 15:42:47.219724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.219652 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" event={"ID":"822b28e8-240d-4cf5-b6de-87900b6c5938","Type":"ContainerStarted","Data":"8cc669f600652554e32560998f1e2684a971806e755a6a93298512e5bf16c1e7"} Nov 03 15:42:47.219912 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.219783 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" event={"ID":"822b28e8-240d-4cf5-b6de-87900b6c5938","Type":"ContainerStarted","Data":"a202cbe01f9d77810e7db9775287c8f8b568622b27fe356db535732b47d65714"} Nov 03 15:42:47.223152 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.223121 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" event={"ID":"cb83d831-2143-4f54-aa05-6fad48074acf","Type":"ContainerStarted","Data":"06e41824471a3346151774600c96971f7cfbfa8643d35aea2699dbc9e43b4ac6"} Nov 03 15:42:47.318360 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.318317 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:47.320411 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.320376 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 15:42:47.320577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.320440 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" podUID="822b28e8-240d-4cf5-b6de-87900b6c5938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 15:42:47.356965 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.356885 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" podStartSLOduration=2.356865276 podStartE2EDuration="2.356865276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:42:47.351458993 +0000 UTC m=+2074.782416803" watchObservedRunningTime="2026-11-03 15:42:47.356865276 +0000 UTC m=+2074.787823084" Nov 03 15:42:47.429066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.429034 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" Nov 03 15:42:47.481401 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.481371 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be2add4-64cd-435f-8cc2-69eca189384f-kube-api-access-ngw6m\" (UniqueName: \"kubernetes.io/projected/8be2add4-64cd-435f-8cc2-69eca189384f-kube-api-access-ngw6m\") pod \"8be2add4-64cd-435f-8cc2-69eca189384f\" (UID: \"8be2add4-64cd-435f-8cc2-69eca189384f\") " Nov 03 15:42:47.481625 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.481608 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be2add4-64cd-435f-8cc2-69eca189384f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be2add4-64cd-435f-8cc2-69eca189384f-maas-api-openshift-ca\") pod \"8be2add4-64cd-435f-8cc2-69eca189384f\" (UID: \"8be2add4-64cd-435f-8cc2-69eca189384f\") " Nov 03 15:42:47.481757 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.481738 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8be2add4-64cd-435f-8cc2-69eca189384f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be2add4-64cd-435f-8cc2-69eca189384f-maas-api-xks-ca\") pod \"8be2add4-64cd-435f-8cc2-69eca189384f\" (UID: \"8be2add4-64cd-435f-8cc2-69eca189384f\") " Nov 03 15:42:47.482075 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.482048 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be2add4-64cd-435f-8cc2-69eca189384f-maas-api-openshift-ca" pod "8be2add4-64cd-435f-8cc2-69eca189384f" (UID: "8be2add4-64cd-435f-8cc2-69eca189384f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:42:47.482331 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.482313 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be2add4-64cd-435f-8cc2-69eca189384f-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:47.483882 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.483857 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be2add4-64cd-435f-8cc2-69eca189384f-kube-api-access-ngw6m" pod "8be2add4-64cd-435f-8cc2-69eca189384f" (UID: "8be2add4-64cd-435f-8cc2-69eca189384f"). InnerVolumeSpecName "kube-api-access-ngw6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:42:47.484010 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.483988 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be2add4-64cd-435f-8cc2-69eca189384f-maas-api-xks-ca" pod "8be2add4-64cd-435f-8cc2-69eca189384f" (UID: "8be2add4-64cd-435f-8cc2-69eca189384f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:42:47.584648 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.584609 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngw6m\" (UniqueName: \"kubernetes.io/projected/8be2add4-64cd-435f-8cc2-69eca189384f-kube-api-access-ngw6m\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:47.584846 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:47.584653 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be2add4-64cd-435f-8cc2-69eca189384f-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:48.232888 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:48.232857 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" Nov 03 15:42:48.232888 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:48.232886 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd" event={"ID":"8be2add4-64cd-435f-8cc2-69eca189384f","Type":"ContainerDied","Data":"a7c6b364e9cc939a64f8f4c99050896d2c52cc1045517258217c0ff36517243d"} Nov 03 15:42:48.233504 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:48.232948 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7c6b364e9cc939a64f8f4c99050896d2c52cc1045517258217c0ff36517243d" Nov 03 15:42:48.255732 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:48.249728 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" event={"ID":"cb83d831-2143-4f54-aa05-6fad48074acf","Type":"ContainerStarted","Data":"778d514436be429a02565ba9528908f6f162ad1c177ca9b21f1ac4d92365691a"} Nov 03 15:42:48.318894 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:48.318858 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 15:42:48.319066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:48.318911 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" podUID="822b28e8-240d-4cf5-b6de-87900b6c5938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 15:42:48.418006 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:48.417915 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" podStartSLOduration=2.41789832 podStartE2EDuration="2.41789832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:42:48.371875452 +0000 UTC m=+2075.802833258" watchObservedRunningTime="2026-11-03 15:42:48.41789832 +0000 UTC m=+2075.848856126" Nov 03 15:42:49.272727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:49.268868 4290 generic.go:356] "Generic (PLEG): container finished" podID="cb83d831-2143-4f54-aa05-6fad48074acf" containerID="778d514436be429a02565ba9528908f6f162ad1c177ca9b21f1ac4d92365691a" exitCode=0 Nov 03 15:42:49.272727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:49.270292 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" event={"ID":"cb83d831-2143-4f54-aa05-6fad48074acf","Type":"ContainerDied","Data":"778d514436be429a02565ba9528908f6f162ad1c177ca9b21f1ac4d92365691a"} Nov 03 15:42:49.318865 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:49.318828 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 15:42:49.319062 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:49.318888 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" podUID="822b28e8-240d-4cf5-b6de-87900b6c5938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 15:42:50.319751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:50.319120 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 15:42:50.319751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:50.319193 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" podUID="822b28e8-240d-4cf5-b6de-87900b6c5938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 15:42:50.511426 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:50.511397 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" Nov 03 15:42:50.628676 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:50.628642 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb83d831-2143-4f54-aa05-6fad48074acf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb83d831-2143-4f54-aa05-6fad48074acf-maas-api-xks-ca\") pod \"cb83d831-2143-4f54-aa05-6fad48074acf\" (UID: \"cb83d831-2143-4f54-aa05-6fad48074acf\") " Nov 03 15:42:50.628908 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:50.628795 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb83d831-2143-4f54-aa05-6fad48074acf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb83d831-2143-4f54-aa05-6fad48074acf-maas-api-openshift-ca\") pod \"cb83d831-2143-4f54-aa05-6fad48074acf\" (UID: \"cb83d831-2143-4f54-aa05-6fad48074acf\") " Nov 03 15:42:50.628908 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:50.628865 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb83d831-2143-4f54-aa05-6fad48074acf-kube-api-access-snzdj\" (UniqueName: \"kubernetes.io/projected/cb83d831-2143-4f54-aa05-6fad48074acf-kube-api-access-snzdj\") pod \"cb83d831-2143-4f54-aa05-6fad48074acf\" (UID: \"cb83d831-2143-4f54-aa05-6fad48074acf\") " Nov 03 15:42:50.629136 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:50.629102 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb83d831-2143-4f54-aa05-6fad48074acf-maas-api-openshift-ca" pod "cb83d831-2143-4f54-aa05-6fad48074acf" (UID: "cb83d831-2143-4f54-aa05-6fad48074acf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:42:50.629385 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:50.629367 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb83d831-2143-4f54-aa05-6fad48074acf-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:50.631132 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:50.631097 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb83d831-2143-4f54-aa05-6fad48074acf-maas-api-xks-ca" pod "cb83d831-2143-4f54-aa05-6fad48074acf" (UID: "cb83d831-2143-4f54-aa05-6fad48074acf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:42:50.631132 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:50.631122 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb83d831-2143-4f54-aa05-6fad48074acf-kube-api-access-snzdj" pod "cb83d831-2143-4f54-aa05-6fad48074acf" (UID: "cb83d831-2143-4f54-aa05-6fad48074acf"). InnerVolumeSpecName "kube-api-access-snzdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:42:50.730796 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:50.730755 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snzdj\" (UniqueName: \"kubernetes.io/projected/cb83d831-2143-4f54-aa05-6fad48074acf-kube-api-access-snzdj\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:50.730796 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:50.730786 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb83d831-2143-4f54-aa05-6fad48074acf-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:51.289318 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:51.289288 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" Nov 03 15:42:51.289507 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:51.289324 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m" event={"ID":"cb83d831-2143-4f54-aa05-6fad48074acf","Type":"ContainerDied","Data":"06e41824471a3346151774600c96971f7cfbfa8643d35aea2699dbc9e43b4ac6"} Nov 03 15:42:51.289507 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:51.289387 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06e41824471a3346151774600c96971f7cfbfa8643d35aea2699dbc9e43b4ac6" Nov 03 15:42:51.318556 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:51.318518 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 15:42:51.318760 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:51.318571 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" podUID="822b28e8-240d-4cf5-b6de-87900b6c5938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 15:42:52.319099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:52.319065 4290 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 15:42:52.319502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:52.319126 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" podUID="822b28e8-240d-4cf5-b6de-87900b6c5938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 15:42:53.323860 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.323831 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:53.324333 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.324141 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:53.324900 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.324875 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:42:53.436543 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.436486 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k"] Nov 03 15:42:53.437844 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.437816 4290 cpu_manager.go:414] "containerMap: removing container" podUID="cb83d831-2143-4f54-aa05-6fad48074acf" containerName="revoke-keys" Nov 03 15:42:53.437844 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.437841 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb83d831-2143-4f54-aa05-6fad48074acf" containerName="revoke-keys" Nov 03 15:42:53.438056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.437860 4290 cpu_manager.go:414] "containerMap: removing container" podUID="8be2add4-64cd-435f-8cc2-69eca189384f" containerName="revoke-keys" Nov 03 15:42:53.438056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.437866 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be2add4-64cd-435f-8cc2-69eca189384f" containerName="revoke-keys" Nov 03 15:42:53.438056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.438007 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb83d831-2143-4f54-aa05-6fad48074acf" containerName="revoke-keys" Nov 03 15:42:53.438056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.438058 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be2add4-64cd-435f-8cc2-69eca189384f" containerName="revoke-keys" Nov 03 15:42:53.442358 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.442331 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" Nov 03 15:42:53.442488 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.442413 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph"] Nov 03 15:42:53.447000 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.446978 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" Nov 03 15:42:53.457613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.457582 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k"] Nov 03 15:42:53.519753 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.517773 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph"] Nov 03 15:42:53.532046 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.532009 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:42:53.532046 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.532032 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:42:53.600887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.600799 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71993bd0-5109-43ca-a5d9-ab525546ced5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b33e18c8-ntc4k\" (UID: \"71993bd0-5109-43ca-a5d9-ab525546ced5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" Nov 03 15:42:53.600887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.600854 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71993bd0-5109-43ca-a5d9-ab525546ced5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b33e18c8-ntc4k\" (UID: \"71993bd0-5109-43ca-a5d9-ab525546ced5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" Nov 03 15:42:53.601066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.601041 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvddl\" (UniqueName: \"kubernetes.io/projected/e0d9bd51-c718-47a2-8aac-1f32315e1592-kube-api-access-bvddl\") pod \"maas-api-revoke-sub-ddeebef3-7zlph\" (UID: \"e0d9bd51-c718-47a2-8aac-1f32315e1592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" Nov 03 15:42:53.601122 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.601102 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d9bd51-c718-47a2-8aac-1f32315e1592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddeebef3-7zlph\" (UID: \"e0d9bd51-c718-47a2-8aac-1f32315e1592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" Nov 03 15:42:53.601243 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.601205 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhkqx\" (UniqueName: \"kubernetes.io/projected/71993bd0-5109-43ca-a5d9-ab525546ced5-kube-api-access-hhkqx\") pod \"maas-api-revoke-sub-b33e18c8-ntc4k\" (UID: \"71993bd0-5109-43ca-a5d9-ab525546ced5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" Nov 03 15:42:53.601369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.601272 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d9bd51-c718-47a2-8aac-1f32315e1592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddeebef3-7zlph\" (UID: \"e0d9bd51-c718-47a2-8aac-1f32315e1592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" Nov 03 15:42:53.702308 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.702271 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71993bd0-5109-43ca-a5d9-ab525546ced5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b33e18c8-ntc4k\" (UID: \"71993bd0-5109-43ca-a5d9-ab525546ced5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" Nov 03 15:42:53.702492 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.702371 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvddl\" (UniqueName: \"kubernetes.io/projected/e0d9bd51-c718-47a2-8aac-1f32315e1592-kube-api-access-bvddl\") pod \"maas-api-revoke-sub-ddeebef3-7zlph\" (UID: \"e0d9bd51-c718-47a2-8aac-1f32315e1592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" Nov 03 15:42:53.702492 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.702415 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d9bd51-c718-47a2-8aac-1f32315e1592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddeebef3-7zlph\" (UID: \"e0d9bd51-c718-47a2-8aac-1f32315e1592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" Nov 03 15:42:53.702492 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.702462 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhkqx\" (UniqueName: \"kubernetes.io/projected/71993bd0-5109-43ca-a5d9-ab525546ced5-kube-api-access-hhkqx\") pod \"maas-api-revoke-sub-b33e18c8-ntc4k\" (UID: \"71993bd0-5109-43ca-a5d9-ab525546ced5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" Nov 03 15:42:53.702678 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.702509 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d9bd51-c718-47a2-8aac-1f32315e1592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddeebef3-7zlph\" (UID: \"e0d9bd51-c718-47a2-8aac-1f32315e1592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" Nov 03 15:42:53.702822 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.702800 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71993bd0-5109-43ca-a5d9-ab525546ced5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b33e18c8-ntc4k\" (UID: \"71993bd0-5109-43ca-a5d9-ab525546ced5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" Nov 03 15:42:53.703534 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.703505 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71993bd0-5109-43ca-a5d9-ab525546ced5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b33e18c8-ntc4k\" (UID: \"71993bd0-5109-43ca-a5d9-ab525546ced5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" Nov 03 15:42:53.703534 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.703521 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d9bd51-c718-47a2-8aac-1f32315e1592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddeebef3-7zlph\" (UID: \"e0d9bd51-c718-47a2-8aac-1f32315e1592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" Nov 03 15:42:53.705254 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.705231 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71993bd0-5109-43ca-a5d9-ab525546ced5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b33e18c8-ntc4k\" (UID: \"71993bd0-5109-43ca-a5d9-ab525546ced5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" Nov 03 15:42:53.706326 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.706303 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d9bd51-c718-47a2-8aac-1f32315e1592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddeebef3-7zlph\" (UID: \"e0d9bd51-c718-47a2-8aac-1f32315e1592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" Nov 03 15:42:53.766968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.766921 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvddl\" (UniqueName: \"kubernetes.io/projected/e0d9bd51-c718-47a2-8aac-1f32315e1592-kube-api-access-bvddl\") pod \"maas-api-revoke-sub-ddeebef3-7zlph\" (UID: \"e0d9bd51-c718-47a2-8aac-1f32315e1592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" Nov 03 15:42:53.768927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.768903 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" Nov 03 15:42:53.769280 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:53.769255 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhkqx\" (UniqueName: \"kubernetes.io/projected/71993bd0-5109-43ca-a5d9-ab525546ced5-kube-api-access-hhkqx\") pod \"maas-api-revoke-sub-b33e18c8-ntc4k\" (UID: \"71993bd0-5109-43ca-a5d9-ab525546ced5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" Nov 03 15:42:54.062114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:54.062080 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" Nov 03 15:42:54.187496 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:54.185883 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph"] Nov 03 15:42:54.317847 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:54.317803 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" event={"ID":"e0d9bd51-c718-47a2-8aac-1f32315e1592","Type":"ContainerStarted","Data":"1a4fc9041af1cd744cdd9528d976f80c1e9fe0f3fd42853886d0d57083ca3817"} Nov 03 15:42:54.465767 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:42:54.465736 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71993bd0_5109_43ca_a5d9_ab525546ced5.slice/crio-e784624d3aaeab21e15725154b91a0cd06bab0b19881537ba5de25b66fdd6397 WatchSource:0}: Error finding container e784624d3aaeab21e15725154b91a0cd06bab0b19881537ba5de25b66fdd6397: Status 404 returned error can't find the container with id e784624d3aaeab21e15725154b91a0cd06bab0b19881537ba5de25b66fdd6397 Nov 03 15:42:54.466721 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:54.466632 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k"] Nov 03 15:42:55.101784 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.101274 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z"] Nov 03 15:42:55.115646 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.115269 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" Nov 03 15:42:55.164731 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.160854 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z"] Nov 03 15:42:55.180853 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.178573 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5cad072f\"" Nov 03 15:42:55.180853 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.179390 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5cad072f-dockercfg-hz5qn\"" Nov 03 15:42:55.211772 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.211739 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6"] Nov 03 15:42:55.217212 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.217186 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" Nov 03 15:42:55.233453 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.233421 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c5ffbd-5f53-447d-b85b-538a4f326c7f-plugins-config-volume\") pod \"payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z\" (UID: \"60c5ffbd-5f53-447d-b85b-538a4f326c7f\") " pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" Nov 03 15:42:55.233657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.233633 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkj96\" (UniqueName: \"kubernetes.io/projected/60c5ffbd-5f53-447d-b85b-538a4f326c7f-kube-api-access-xkj96\") pod \"payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z\" (UID: \"60c5ffbd-5f53-447d-b85b-538a4f326c7f\") " pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" Nov 03 15:42:55.271315 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.271282 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6"] Nov 03 15:42:55.330174 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.330133 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" event={"ID":"71993bd0-5109-43ca-a5d9-ab525546ced5","Type":"ContainerStarted","Data":"da3ecf52f44466ca778d7ad1d0fa7a450de8184146404bc3c777831a7977860a"} Nov 03 15:42:55.330378 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.330196 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" event={"ID":"71993bd0-5109-43ca-a5d9-ab525546ced5","Type":"ContainerStarted","Data":"e784624d3aaeab21e15725154b91a0cd06bab0b19881537ba5de25b66fdd6397"} Nov 03 15:42:55.332142 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.332110 4290 generic.go:356] "Generic (PLEG): container finished" podID="e0d9bd51-c718-47a2-8aac-1f32315e1592" containerID="4d64765ecc7d34a1049d0c418b4c6829d990ced5d985a04a79fb6bf063450ab9" exitCode=0 Nov 03 15:42:55.332256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.332190 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" event={"ID":"e0d9bd51-c718-47a2-8aac-1f32315e1592","Type":"ContainerDied","Data":"4d64765ecc7d34a1049d0c418b4c6829d990ced5d985a04a79fb6bf063450ab9"} Nov 03 15:42:55.335123 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.335101 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c5ffbd-5f53-447d-b85b-538a4f326c7f-plugins-config-volume\") pod \"payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z\" (UID: \"60c5ffbd-5f53-447d-b85b-538a4f326c7f\") " pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" Nov 03 15:42:55.335246 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.335228 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkj96\" (UniqueName: \"kubernetes.io/projected/60c5ffbd-5f53-447d-b85b-538a4f326c7f-kube-api-access-xkj96\") pod \"payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z\" (UID: \"60c5ffbd-5f53-447d-b85b-538a4f326c7f\") " pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" Nov 03 15:42:55.335304 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.335289 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvzjw\" (UniqueName: \"kubernetes.io/projected/1b6b120a-3b26-4f20-a145-ccdc060e6ea8-kube-api-access-wvzjw\") pod \"payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6\" (UID: \"1b6b120a-3b26-4f20-a145-ccdc060e6ea8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" Nov 03 15:42:55.335479 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.335460 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6b120a-3b26-4f20-a145-ccdc060e6ea8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6\" (UID: \"1b6b120a-3b26-4f20-a145-ccdc060e6ea8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" Nov 03 15:42:55.335847 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.335830 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c5ffbd-5f53-447d-b85b-538a4f326c7f-plugins-config-volume\") pod \"payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z\" (UID: \"60c5ffbd-5f53-447d-b85b-538a4f326c7f\") " pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" Nov 03 15:42:55.396823 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.396791 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkj96\" (UniqueName: \"kubernetes.io/projected/60c5ffbd-5f53-447d-b85b-538a4f326c7f-kube-api-access-xkj96\") pod \"payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z\" (UID: \"60c5ffbd-5f53-447d-b85b-538a4f326c7f\") " pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" Nov 03 15:42:55.436465 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.436427 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzjw\" (UniqueName: \"kubernetes.io/projected/1b6b120a-3b26-4f20-a145-ccdc060e6ea8-kube-api-access-wvzjw\") pod \"payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6\" (UID: \"1b6b120a-3b26-4f20-a145-ccdc060e6ea8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" Nov 03 15:42:55.436827 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.436803 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6b120a-3b26-4f20-a145-ccdc060e6ea8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6\" (UID: \"1b6b120a-3b26-4f20-a145-ccdc060e6ea8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" Nov 03 15:42:55.437619 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.437596 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6b120a-3b26-4f20-a145-ccdc060e6ea8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6\" (UID: \"1b6b120a-3b26-4f20-a145-ccdc060e6ea8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" Nov 03 15:42:55.496523 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.496453 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" podStartSLOduration=2.496434945 podStartE2EDuration="2.496434945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:42:55.49194279 +0000 UTC m=+2082.922900596" watchObservedRunningTime="2026-11-03 15:42:55.496434945 +0000 UTC m=+2082.927392754" Nov 03 15:42:55.507857 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.507823 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvzjw\" (UniqueName: \"kubernetes.io/projected/1b6b120a-3b26-4f20-a145-ccdc060e6ea8-kube-api-access-wvzjw\") pod \"payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6\" (UID: \"1b6b120a-3b26-4f20-a145-ccdc060e6ea8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" Nov 03 15:42:55.512590 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.512562 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" Nov 03 15:42:55.534213 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.534186 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" Nov 03 15:42:55.904353 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:42:55.904315 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c5ffbd_5f53_447d_b85b_538a4f326c7f.slice/crio-0393f3ab73cd10592603ae50632766bf027eb528e09763890b919c89f84af109 WatchSource:0}: Error finding container 0393f3ab73cd10592603ae50632766bf027eb528e09763890b919c89f84af109: Status 404 returned error can't find the container with id 0393f3ab73cd10592603ae50632766bf027eb528e09763890b919c89f84af109 Nov 03 15:42:55.905925 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.905897 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z"] Nov 03 15:42:55.956095 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:55.956067 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6"] Nov 03 15:42:55.959283 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:42:55.959248 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6b120a_3b26_4f20_a145_ccdc060e6ea8.slice/crio-3074332888f9b784fa6a66a308baed7f10965b6de5a8c75514dca73d93aa8d8e WatchSource:0}: Error finding container 3074332888f9b784fa6a66a308baed7f10965b6de5a8c75514dca73d93aa8d8e: Status 404 returned error can't find the container with id 3074332888f9b784fa6a66a308baed7f10965b6de5a8c75514dca73d93aa8d8e Nov 03 15:42:56.342590 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.342498 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" event={"ID":"60c5ffbd-5f53-447d-b85b-538a4f326c7f","Type":"ContainerStarted","Data":"57b52786c7bb5a61e90615777968eabe0cea1a0fb99bb0c43aa0336ad50689c1"} Nov 03 15:42:56.342590 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.342555 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" event={"ID":"60c5ffbd-5f53-447d-b85b-538a4f326c7f","Type":"ContainerStarted","Data":"0393f3ab73cd10592603ae50632766bf027eb528e09763890b919c89f84af109"} Nov 03 15:42:56.342590 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.342582 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" Nov 03 15:42:56.344472 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.344437 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" event={"ID":"1b6b120a-3b26-4f20-a145-ccdc060e6ea8","Type":"ContainerStarted","Data":"f97e37a45ef5f6ad9505165cfc538b7183c7086e8cffd360092c5201307713c5"} Nov 03 15:42:56.344601 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.344499 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" Nov 03 15:42:56.344601 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.344522 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" event={"ID":"1b6b120a-3b26-4f20-a145-ccdc060e6ea8","Type":"ContainerStarted","Data":"3074332888f9b784fa6a66a308baed7f10965b6de5a8c75514dca73d93aa8d8e"} Nov 03 15:42:56.346169 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.346144 4290 generic.go:356] "Generic (PLEG): container finished" podID="71993bd0-5109-43ca-a5d9-ab525546ced5" containerID="da3ecf52f44466ca778d7ad1d0fa7a450de8184146404bc3c777831a7977860a" exitCode=0 Nov 03 15:42:56.346277 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.346232 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" event={"ID":"71993bd0-5109-43ca-a5d9-ab525546ced5","Type":"ContainerDied","Data":"da3ecf52f44466ca778d7ad1d0fa7a450de8184146404bc3c777831a7977860a"} Nov 03 15:42:56.463222 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.463158 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" podStartSLOduration=1.463141379 podStartE2EDuration="1.463141379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:42:56.461765409 +0000 UTC m=+2083.892723218" watchObservedRunningTime="2026-11-03 15:42:56.463141379 +0000 UTC m=+2083.894099186" Nov 03 15:42:56.498681 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.498646 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" Nov 03 15:42:56.559510 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.559475 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d9bd51-c718-47a2-8aac-1f32315e1592-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d9bd51-c718-47a2-8aac-1f32315e1592-maas-api-openshift-ca\") pod \"e0d9bd51-c718-47a2-8aac-1f32315e1592\" (UID: \"e0d9bd51-c718-47a2-8aac-1f32315e1592\") " Nov 03 15:42:56.559683 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.559655 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d9bd51-c718-47a2-8aac-1f32315e1592-kube-api-access-bvddl\" (UniqueName: \"kubernetes.io/projected/e0d9bd51-c718-47a2-8aac-1f32315e1592-kube-api-access-bvddl\") pod \"e0d9bd51-c718-47a2-8aac-1f32315e1592\" (UID: \"e0d9bd51-c718-47a2-8aac-1f32315e1592\") " Nov 03 15:42:56.559850 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.559829 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0d9bd51-c718-47a2-8aac-1f32315e1592-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d9bd51-c718-47a2-8aac-1f32315e1592-maas-api-xks-ca\") pod \"e0d9bd51-c718-47a2-8aac-1f32315e1592\" (UID: \"e0d9bd51-c718-47a2-8aac-1f32315e1592\") " Nov 03 15:42:56.560237 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.560017 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d9bd51-c718-47a2-8aac-1f32315e1592-maas-api-openshift-ca" pod "e0d9bd51-c718-47a2-8aac-1f32315e1592" (UID: "e0d9bd51-c718-47a2-8aac-1f32315e1592"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:42:56.560736 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.560662 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d9bd51-c718-47a2-8aac-1f32315e1592-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:56.562118 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.562092 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d9bd51-c718-47a2-8aac-1f32315e1592-kube-api-access-bvddl" pod "e0d9bd51-c718-47a2-8aac-1f32315e1592" (UID: "e0d9bd51-c718-47a2-8aac-1f32315e1592"). InnerVolumeSpecName "kube-api-access-bvddl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:42:56.562204 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.562155 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d9bd51-c718-47a2-8aac-1f32315e1592-maas-api-xks-ca" pod "e0d9bd51-c718-47a2-8aac-1f32315e1592" (UID: "e0d9bd51-c718-47a2-8aac-1f32315e1592"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:42:56.662600 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.662561 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvddl\" (UniqueName: \"kubernetes.io/projected/e0d9bd51-c718-47a2-8aac-1f32315e1592-kube-api-access-bvddl\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:56.662600 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.662604 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d9bd51-c718-47a2-8aac-1f32315e1592-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:56.697168 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:56.697118 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" podStartSLOduration=1.697102883 podStartE2EDuration="1.697102883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:42:56.694024436 +0000 UTC m=+2084.124982243" watchObservedRunningTime="2026-11-03 15:42:56.697102883 +0000 UTC m=+2084.128060689" Nov 03 15:42:57.357800 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:57.357760 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" event={"ID":"e0d9bd51-c718-47a2-8aac-1f32315e1592","Type":"ContainerDied","Data":"1a4fc9041af1cd744cdd9528d976f80c1e9fe0f3fd42853886d0d57083ca3817"} Nov 03 15:42:57.357800 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:57.357786 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph" Nov 03 15:42:57.358043 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:57.357825 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a4fc9041af1cd744cdd9528d976f80c1e9fe0f3fd42853886d0d57083ca3817" Nov 03 15:42:57.545340 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:57.545308 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" Nov 03 15:42:57.679230 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:57.679191 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71993bd0-5109-43ca-a5d9-ab525546ced5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71993bd0-5109-43ca-a5d9-ab525546ced5-maas-api-openshift-ca\") pod \"71993bd0-5109-43ca-a5d9-ab525546ced5\" (UID: \"71993bd0-5109-43ca-a5d9-ab525546ced5\") " Nov 03 15:42:57.679468 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:57.679449 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71993bd0-5109-43ca-a5d9-ab525546ced5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71993bd0-5109-43ca-a5d9-ab525546ced5-maas-api-xks-ca\") pod \"71993bd0-5109-43ca-a5d9-ab525546ced5\" (UID: \"71993bd0-5109-43ca-a5d9-ab525546ced5\") " Nov 03 15:42:57.679591 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:57.679504 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71993bd0-5109-43ca-a5d9-ab525546ced5-kube-api-access-hhkqx\" (UniqueName: \"kubernetes.io/projected/71993bd0-5109-43ca-a5d9-ab525546ced5-kube-api-access-hhkqx\") pod \"71993bd0-5109-43ca-a5d9-ab525546ced5\" (UID: \"71993bd0-5109-43ca-a5d9-ab525546ced5\") " Nov 03 15:42:57.679665 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:57.679592 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71993bd0-5109-43ca-a5d9-ab525546ced5-maas-api-openshift-ca" pod "71993bd0-5109-43ca-a5d9-ab525546ced5" (UID: "71993bd0-5109-43ca-a5d9-ab525546ced5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:42:57.680489 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:57.680457 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71993bd0-5109-43ca-a5d9-ab525546ced5-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:57.682187 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:57.682158 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71993bd0-5109-43ca-a5d9-ab525546ced5-kube-api-access-hhkqx" pod "71993bd0-5109-43ca-a5d9-ab525546ced5" (UID: "71993bd0-5109-43ca-a5d9-ab525546ced5"). InnerVolumeSpecName "kube-api-access-hhkqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:42:57.685150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:57.685127 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71993bd0-5109-43ca-a5d9-ab525546ced5-maas-api-xks-ca" pod "71993bd0-5109-43ca-a5d9-ab525546ced5" (UID: "71993bd0-5109-43ca-a5d9-ab525546ced5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:42:57.782292 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:57.782245 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71993bd0-5109-43ca-a5d9-ab525546ced5-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:57.782292 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:57.782277 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhkqx\" (UniqueName: \"kubernetes.io/projected/71993bd0-5109-43ca-a5d9-ab525546ced5-kube-api-access-hhkqx\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:42:58.372614 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:58.372585 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" Nov 03 15:42:58.372614 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:58.372602 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k" event={"ID":"71993bd0-5109-43ca-a5d9-ab525546ced5","Type":"ContainerDied","Data":"e784624d3aaeab21e15725154b91a0cd06bab0b19881537ba5de25b66fdd6397"} Nov 03 15:42:58.372869 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:42:58.372662 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e784624d3aaeab21e15725154b91a0cd06bab0b19881537ba5de25b66fdd6397" Nov 03 15:43:07.361270 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:07.361235 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" Nov 03 15:43:07.361794 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:07.361311 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" Nov 03 15:43:08.917747 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:43:08.911910 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:43:13.172400 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:13.172368 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:43:13.175719 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:13.175680 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:43:13.184209 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:13.184184 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:43:13.187759 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:13.187640 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:43:28.407400 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.407321 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6"] Nov 03 15:43:28.408119 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.408104 4290 cpu_manager.go:414] "containerMap: removing container" podUID="71993bd0-5109-43ca-a5d9-ab525546ced5" containerName="revoke-keys" Nov 03 15:43:28.408164 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.408121 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71993bd0-5109-43ca-a5d9-ab525546ced5" containerName="revoke-keys" Nov 03 15:43:28.408164 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.408153 4290 cpu_manager.go:414] "containerMap: removing container" podUID="e0d9bd51-c718-47a2-8aac-1f32315e1592" containerName="revoke-keys" Nov 03 15:43:28.408164 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.408158 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d9bd51-c718-47a2-8aac-1f32315e1592" containerName="revoke-keys" Nov 03 15:43:28.408256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.408246 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d9bd51-c718-47a2-8aac-1f32315e1592" containerName="revoke-keys" Nov 03 15:43:28.408289 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.408262 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="71993bd0-5109-43ca-a5d9-ab525546ced5" containerName="revoke-keys" Nov 03 15:43:28.413214 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.413195 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" Nov 03 15:43:28.466682 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.466643 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:43:28.516764 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.516737 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:43:28.525841 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.525806 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6"] Nov 03 15:43:28.529762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.529692 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764b9a1a-dbcf-4875-8339-ea19b0213e40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6\" (UID: \"764b9a1a-dbcf-4875-8339-ea19b0213e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" Nov 03 15:43:28.529914 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.529892 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9m4x\" (UniqueName: \"kubernetes.io/projected/764b9a1a-dbcf-4875-8339-ea19b0213e40-kube-api-access-q9m4x\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6\" (UID: \"764b9a1a-dbcf-4875-8339-ea19b0213e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" Nov 03 15:43:28.530016 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.530001 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764b9a1a-dbcf-4875-8339-ea19b0213e40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6\" (UID: \"764b9a1a-dbcf-4875-8339-ea19b0213e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" Nov 03 15:43:28.631181 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.631147 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9m4x\" (UniqueName: \"kubernetes.io/projected/764b9a1a-dbcf-4875-8339-ea19b0213e40-kube-api-access-q9m4x\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6\" (UID: \"764b9a1a-dbcf-4875-8339-ea19b0213e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" Nov 03 15:43:28.631381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.631230 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764b9a1a-dbcf-4875-8339-ea19b0213e40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6\" (UID: \"764b9a1a-dbcf-4875-8339-ea19b0213e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" Nov 03 15:43:28.631381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.631343 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764b9a1a-dbcf-4875-8339-ea19b0213e40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6\" (UID: \"764b9a1a-dbcf-4875-8339-ea19b0213e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" Nov 03 15:43:28.632010 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.631984 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764b9a1a-dbcf-4875-8339-ea19b0213e40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6\" (UID: \"764b9a1a-dbcf-4875-8339-ea19b0213e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" Nov 03 15:43:28.633767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.633740 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764b9a1a-dbcf-4875-8339-ea19b0213e40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6\" (UID: \"764b9a1a-dbcf-4875-8339-ea19b0213e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" Nov 03 15:43:28.694512 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.694442 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9m4x\" (UniqueName: \"kubernetes.io/projected/764b9a1a-dbcf-4875-8339-ea19b0213e40-kube-api-access-q9m4x\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6\" (UID: \"764b9a1a-dbcf-4875-8339-ea19b0213e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" Nov 03 15:43:28.727674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:28.727650 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" Nov 03 15:43:29.125098 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:43:29.125068 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764b9a1a_dbcf_4875_8339_ea19b0213e40.slice/crio-bfa69c8e3d6976a46b6842c684380ea1408092dfbfcb34ddbe0e92897a8a0ef2 WatchSource:0}: Error finding container bfa69c8e3d6976a46b6842c684380ea1408092dfbfcb34ddbe0e92897a8a0ef2: Status 404 returned error can't find the container with id bfa69c8e3d6976a46b6842c684380ea1408092dfbfcb34ddbe0e92897a8a0ef2 Nov 03 15:43:29.126295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:29.126270 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6"] Nov 03 15:43:29.730848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:29.730751 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" event={"ID":"764b9a1a-dbcf-4875-8339-ea19b0213e40","Type":"ContainerStarted","Data":"fa89ea23c33191bea2165c6448206654fba39b5263f4e38f7bf13b24c5004132"} Nov 03 15:43:29.730848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:29.730810 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" event={"ID":"764b9a1a-dbcf-4875-8339-ea19b0213e40","Type":"ContainerStarted","Data":"bfa69c8e3d6976a46b6842c684380ea1408092dfbfcb34ddbe0e92897a8a0ef2"} Nov 03 15:43:29.873646 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:29.873584 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" podStartSLOduration=1.873567123 podStartE2EDuration="1.873567123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:43:29.868193115 +0000 UTC m=+2117.299150934" watchObservedRunningTime="2026-11-03 15:43:29.873567123 +0000 UTC m=+2117.304524959" Nov 03 15:43:30.741114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:30.741060 4290 generic.go:356] "Generic (PLEG): container finished" podID="764b9a1a-dbcf-4875-8339-ea19b0213e40" containerID="fa89ea23c33191bea2165c6448206654fba39b5263f4e38f7bf13b24c5004132" exitCode=0 Nov 03 15:43:30.741529 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:30.741206 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" event={"ID":"764b9a1a-dbcf-4875-8339-ea19b0213e40","Type":"ContainerDied","Data":"fa89ea23c33191bea2165c6448206654fba39b5263f4e38f7bf13b24c5004132"} Nov 03 15:43:31.927035 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:31.927007 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" Nov 03 15:43:31.979276 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:31.979241 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/764b9a1a-dbcf-4875-8339-ea19b0213e40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764b9a1a-dbcf-4875-8339-ea19b0213e40-maas-api-openshift-ca\") pod \"764b9a1a-dbcf-4875-8339-ea19b0213e40\" (UID: \"764b9a1a-dbcf-4875-8339-ea19b0213e40\") " Nov 03 15:43:31.979457 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:31.979341 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/764b9a1a-dbcf-4875-8339-ea19b0213e40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764b9a1a-dbcf-4875-8339-ea19b0213e40-maas-api-xks-ca\") pod \"764b9a1a-dbcf-4875-8339-ea19b0213e40\" (UID: \"764b9a1a-dbcf-4875-8339-ea19b0213e40\") " Nov 03 15:43:31.979522 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:31.979493 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764b9a1a-dbcf-4875-8339-ea19b0213e40-kube-api-access-q9m4x\" (UniqueName: \"kubernetes.io/projected/764b9a1a-dbcf-4875-8339-ea19b0213e40-kube-api-access-q9m4x\") pod \"764b9a1a-dbcf-4875-8339-ea19b0213e40\" (UID: \"764b9a1a-dbcf-4875-8339-ea19b0213e40\") " Nov 03 15:43:31.979652 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:31.979616 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764b9a1a-dbcf-4875-8339-ea19b0213e40-maas-api-openshift-ca" pod "764b9a1a-dbcf-4875-8339-ea19b0213e40" (UID: "764b9a1a-dbcf-4875-8339-ea19b0213e40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:43:31.980067 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:31.980045 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764b9a1a-dbcf-4875-8339-ea19b0213e40-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:31.982055 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:31.982031 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764b9a1a-dbcf-4875-8339-ea19b0213e40-kube-api-access-q9m4x" pod "764b9a1a-dbcf-4875-8339-ea19b0213e40" (UID: "764b9a1a-dbcf-4875-8339-ea19b0213e40"). InnerVolumeSpecName "kube-api-access-q9m4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:43:31.982170 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:31.982123 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764b9a1a-dbcf-4875-8339-ea19b0213e40-maas-api-xks-ca" pod "764b9a1a-dbcf-4875-8339-ea19b0213e40" (UID: "764b9a1a-dbcf-4875-8339-ea19b0213e40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:43:32.081891 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:32.081806 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764b9a1a-dbcf-4875-8339-ea19b0213e40-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:32.081891 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:32.081847 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9m4x\" (UniqueName: \"kubernetes.io/projected/764b9a1a-dbcf-4875-8339-ea19b0213e40-kube-api-access-q9m4x\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:32.767721 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:32.766935 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" Nov 03 15:43:32.767721 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:32.766967 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6" event={"ID":"764b9a1a-dbcf-4875-8339-ea19b0213e40","Type":"ContainerDied","Data":"bfa69c8e3d6976a46b6842c684380ea1408092dfbfcb34ddbe0e92897a8a0ef2"} Nov 03 15:43:32.767721 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:32.767030 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfa69c8e3d6976a46b6842c684380ea1408092dfbfcb34ddbe0e92897a8a0ef2" Nov 03 15:43:38.254175 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:38.252931 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb"] Nov 03 15:43:38.254175 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:38.253173 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" podUID="6f7d95e8-3d59-481b-bdfd-e33e6854d805" containerName="istio-proxy" containerID="cri-o://864e8c064cb273491dfa65c3742afd06ae4ff405777a47e186c76d6fe6d02907" gracePeriod=30 Nov 03 15:43:38.703963 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:43:38.703922 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:43:39.047609 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:39.047529 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g"] Nov 03 15:43:39.047857 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:39.047813 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" podUID="5a896388-6490-47c0-9eed-5c81fde73165" containerName="istio-proxy" containerID="cri-o://c6065e0fedecb88b3e2893d8dcfeabc040d6e02612fa8c72c99332991a862f68" gracePeriod=30 Nov 03 15:43:39.372347 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:39.372301 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g"] Nov 03 15:43:39.372841 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:39.372517 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" podUID="0a9f66f4-3203-4f72-bce5-c94005f38c35" containerName="istio-proxy" containerID="cri-o://16424b11aec3eec54c63a4117be2d547a01cdc6d4a18fe0d79eb14831674354b" gracePeriod=30 Nov 03 15:43:39.914561 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:39.914521 4290 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:43:39.914763 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:39.914584 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" podUID="6f7d95e8-3d59-481b-bdfd-e33e6854d805" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:43:40.485417 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:40.485380 4290 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:43:40.485839 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:40.485446 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" podUID="0a9f66f4-3203-4f72-bce5-c94005f38c35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:43:41.307169 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:41.307136 4290 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:43:41.307334 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:41.307188 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" podUID="5a896388-6490-47c0-9eed-5c81fde73165" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:43:43.535332 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.535304 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:43:43.648094 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.648068 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7d95e8-3d59-481b-bdfd-e33e6854d805-kube-api-access-fd77q\" (UniqueName: \"kubernetes.io/projected/6f7d95e8-3d59-481b-bdfd-e33e6854d805-kube-api-access-fd77q\") pod \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " Nov 03 15:43:43.648224 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.648105 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-workload-socket\") pod \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " Nov 03 15:43:43.648224 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.648144 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-envoy\") pod \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " Nov 03 15:43:43.648311 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.648266 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-workload-certs\") pod \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " Nov 03 15:43:43.648355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.648329 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-credential-socket\") pod \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " Nov 03 15:43:43.648392 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.648360 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-workload-socket" pod "6f7d95e8-3d59-481b-bdfd-e33e6854d805" (UID: "6f7d95e8-3d59-481b-bdfd-e33e6854d805"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:43.648435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.648389 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-token\" (UniqueName: \"kubernetes.io/projected/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-token\") pod \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " Nov 03 15:43:43.648435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.648424 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-data\") pod \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " Nov 03 15:43:43.648535 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.648469 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-podinfo\") pod \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " Nov 03 15:43:43.648594 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.648521 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-workload-certs" pod "6f7d95e8-3d59-481b-bdfd-e33e6854d805" (UID: "6f7d95e8-3d59-481b-bdfd-e33e6854d805"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:43.648594 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.648543 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istiod-ca-cert\") pod \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\" (UID: \"6f7d95e8-3d59-481b-bdfd-e33e6854d805\") " Nov 03 15:43:43.648717 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.648625 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-credential-socket" pod "6f7d95e8-3d59-481b-bdfd-e33e6854d805" (UID: "6f7d95e8-3d59-481b-bdfd-e33e6854d805"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:43.648717 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.648644 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-data" pod "6f7d95e8-3d59-481b-bdfd-e33e6854d805" (UID: "6f7d95e8-3d59-481b-bdfd-e33e6854d805"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:43.649033 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.649010 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istiod-ca-cert" pod "6f7d95e8-3d59-481b-bdfd-e33e6854d805" (UID: "6f7d95e8-3d59-481b-bdfd-e33e6854d805"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:43:43.649602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.649577 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.649726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.649606 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.649726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.649622 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.649726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.649636 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.649726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.649650 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.650763 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.650735 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7d95e8-3d59-481b-bdfd-e33e6854d805-kube-api-access-fd77q" pod "6f7d95e8-3d59-481b-bdfd-e33e6854d805" (UID: "6f7d95e8-3d59-481b-bdfd-e33e6854d805"). InnerVolumeSpecName "kube-api-access-fd77q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:43:43.650763 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.650749 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-envoy" pod "6f7d95e8-3d59-481b-bdfd-e33e6854d805" (UID: "6f7d95e8-3d59-481b-bdfd-e33e6854d805"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:43.650919 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.650812 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-token" pod "6f7d95e8-3d59-481b-bdfd-e33e6854d805" (UID: "6f7d95e8-3d59-481b-bdfd-e33e6854d805"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:43:43.651002 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.650985 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-podinfo" pod "6f7d95e8-3d59-481b-bdfd-e33e6854d805" (UID: "6f7d95e8-3d59-481b-bdfd-e33e6854d805"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:43:43.751618 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.751575 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.751618 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.751609 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.751927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.751628 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd77q\" (UniqueName: \"kubernetes.io/projected/6f7d95e8-3d59-481b-bdfd-e33e6854d805-kube-api-access-fd77q\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.751927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.751642 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7d95e8-3d59-481b-bdfd-e33e6854d805-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.868311 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.868271 4290 generic.go:356] "Generic (PLEG): container finished" podID="6f7d95e8-3d59-481b-bdfd-e33e6854d805" containerID="864e8c064cb273491dfa65c3742afd06ae4ff405777a47e186c76d6fe6d02907" exitCode=0 Nov 03 15:43:43.868519 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.868323 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" Nov 03 15:43:43.868519 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.868360 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" event={"ID":"6f7d95e8-3d59-481b-bdfd-e33e6854d805","Type":"ContainerDied","Data":"864e8c064cb273491dfa65c3742afd06ae4ff405777a47e186c76d6fe6d02907"} Nov 03 15:43:43.868519 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.868420 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb" event={"ID":"6f7d95e8-3d59-481b-bdfd-e33e6854d805","Type":"ContainerDied","Data":"69882af52ff8edf4a6d159da2eef8ba5088ffb7f1c0ef19d15058b2fc563b1d2"} Nov 03 15:43:43.868519 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.868448 4290 scope.go:122] "RemoveContainer" containerID="864e8c064cb273491dfa65c3742afd06ae4ff405777a47e186c76d6fe6d02907" Nov 03 15:43:43.883904 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.883877 4290 scope.go:122] "RemoveContainer" containerID="864e8c064cb273491dfa65c3742afd06ae4ff405777a47e186c76d6fe6d02907" Nov 03 15:43:43.884325 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:43:43.884302 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864e8c064cb273491dfa65c3742afd06ae4ff405777a47e186c76d6fe6d02907\": container with ID starting with 864e8c064cb273491dfa65c3742afd06ae4ff405777a47e186c76d6fe6d02907 not found: ID does not exist" containerID="864e8c064cb273491dfa65c3742afd06ae4ff405777a47e186c76d6fe6d02907" Nov 03 15:43:43.884436 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:43.884334 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864e8c064cb273491dfa65c3742afd06ae4ff405777a47e186c76d6fe6d02907"} err="failed to get container status \"864e8c064cb273491dfa65c3742afd06ae4ff405777a47e186c76d6fe6d02907\": rpc error: code = NotFound desc = could not find container \"864e8c064cb273491dfa65c3742afd06ae4ff405777a47e186c76d6fe6d02907\": container with ID starting with 864e8c064cb273491dfa65c3742afd06ae4ff405777a47e186c76d6fe6d02907 not found: ID does not exist" Nov 03 15:43:44.039774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.039669 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb"] Nov 03 15:43:44.047404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.047376 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-d88xb"] Nov 03 15:43:44.324584 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.324554 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:43:44.469203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.466829 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-credential-socket\") pod \"5a896388-6490-47c0-9eed-5c81fde73165\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " Nov 03 15:43:44.469203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.466902 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a896388-6490-47c0-9eed-5c81fde73165-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a896388-6490-47c0-9eed-5c81fde73165-istio-podinfo\") pod \"5a896388-6490-47c0-9eed-5c81fde73165\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " Nov 03 15:43:44.469203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.466937 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a896388-6490-47c0-9eed-5c81fde73165-istio-token\" (UniqueName: \"kubernetes.io/projected/5a896388-6490-47c0-9eed-5c81fde73165-istio-token\") pod \"5a896388-6490-47c0-9eed-5c81fde73165\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " Nov 03 15:43:44.469203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.466992 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-workload-certs\") pod \"5a896388-6490-47c0-9eed-5c81fde73165\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " Nov 03 15:43:44.469203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.467058 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-istio-envoy\") pod \"5a896388-6490-47c0-9eed-5c81fde73165\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " Nov 03 15:43:44.469203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.467096 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-credential-socket" pod "5a896388-6490-47c0-9eed-5c81fde73165" (UID: "5a896388-6490-47c0-9eed-5c81fde73165"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:44.469203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.467138 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a896388-6490-47c0-9eed-5c81fde73165-kube-api-access-f7dkh\" (UniqueName: \"kubernetes.io/projected/5a896388-6490-47c0-9eed-5c81fde73165-kube-api-access-f7dkh\") pod \"5a896388-6490-47c0-9eed-5c81fde73165\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " Nov 03 15:43:44.469203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.467255 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a896388-6490-47c0-9eed-5c81fde73165-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a896388-6490-47c0-9eed-5c81fde73165-istiod-ca-cert\") pod \"5a896388-6490-47c0-9eed-5c81fde73165\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " Nov 03 15:43:44.469203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.467313 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-istio-data\") pod \"5a896388-6490-47c0-9eed-5c81fde73165\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " Nov 03 15:43:44.469203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.467342 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-workload-socket\") pod \"5a896388-6490-47c0-9eed-5c81fde73165\" (UID: \"5a896388-6490-47c0-9eed-5c81fde73165\") " Nov 03 15:43:44.469203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.468005 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.469203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.468247 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-workload-socket" pod "5a896388-6490-47c0-9eed-5c81fde73165" (UID: "5a896388-6490-47c0-9eed-5c81fde73165"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:44.469203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.468553 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a896388-6490-47c0-9eed-5c81fde73165-istiod-ca-cert" pod "5a896388-6490-47c0-9eed-5c81fde73165" (UID: "5a896388-6490-47c0-9eed-5c81fde73165"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:43:44.469203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.468820 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-istio-data" pod "5a896388-6490-47c0-9eed-5c81fde73165" (UID: "5a896388-6490-47c0-9eed-5c81fde73165"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:44.469203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.469002 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-workload-certs" pod "5a896388-6490-47c0-9eed-5c81fde73165" (UID: "5a896388-6490-47c0-9eed-5c81fde73165"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:44.476866 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.471057 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a896388-6490-47c0-9eed-5c81fde73165-istio-token" pod "5a896388-6490-47c0-9eed-5c81fde73165" (UID: "5a896388-6490-47c0-9eed-5c81fde73165"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:43:44.476866 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.471652 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-istio-envoy" pod "5a896388-6490-47c0-9eed-5c81fde73165" (UID: "5a896388-6490-47c0-9eed-5c81fde73165"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:44.476866 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.473838 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a896388-6490-47c0-9eed-5c81fde73165-istio-podinfo" pod "5a896388-6490-47c0-9eed-5c81fde73165" (UID: "5a896388-6490-47c0-9eed-5c81fde73165"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:43:44.477673 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.477644 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a896388-6490-47c0-9eed-5c81fde73165-kube-api-access-f7dkh" pod "5a896388-6490-47c0-9eed-5c81fde73165" (UID: "5a896388-6490-47c0-9eed-5c81fde73165"). InnerVolumeSpecName "kube-api-access-f7dkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:43:44.569847 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.569813 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a896388-6490-47c0-9eed-5c81fde73165-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.569847 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.569846 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.570261 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.569857 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.570261 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.569866 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a896388-6490-47c0-9eed-5c81fde73165-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.570261 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.569874 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a896388-6490-47c0-9eed-5c81fde73165-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.570261 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.569897 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.570261 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.569905 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a896388-6490-47c0-9eed-5c81fde73165-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.570261 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.569915 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7dkh\" (UniqueName: \"kubernetes.io/projected/5a896388-6490-47c0-9eed-5c81fde73165-kube-api-access-f7dkh\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.651401 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.651374 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:43:44.680208 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.680179 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f7d95e8-3d59-481b-bdfd-e33e6854d805" path="/var/lib/kubelet/pods/6f7d95e8-3d59-481b-bdfd-e33e6854d805/volumes" Nov 03 15:43:44.772615 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.772569 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-podinfo\") pod \"0a9f66f4-3203-4f72-bce5-c94005f38c35\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " Nov 03 15:43:44.773011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.772631 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9f66f4-3203-4f72-bce5-c94005f38c35-kube-api-access-8hf7w\" (UniqueName: \"kubernetes.io/projected/0a9f66f4-3203-4f72-bce5-c94005f38c35-kube-api-access-8hf7w\") pod \"0a9f66f4-3203-4f72-bce5-c94005f38c35\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " Nov 03 15:43:44.773011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.772684 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-workload-certs\") pod \"0a9f66f4-3203-4f72-bce5-c94005f38c35\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " Nov 03 15:43:44.773011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.772861 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-data\") pod \"0a9f66f4-3203-4f72-bce5-c94005f38c35\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " Nov 03 15:43:44.773011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.772986 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9f66f4-3203-4f72-bce5-c94005f38c35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f66f4-3203-4f72-bce5-c94005f38c35-istiod-ca-cert\") pod \"0a9f66f4-3203-4f72-bce5-c94005f38c35\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " Nov 03 15:43:44.773296 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.773035 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-token\") pod \"0a9f66f4-3203-4f72-bce5-c94005f38c35\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " Nov 03 15:43:44.773296 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.773093 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-credential-socket\") pod \"0a9f66f4-3203-4f72-bce5-c94005f38c35\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " Nov 03 15:43:44.773296 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.773132 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-envoy\") pod \"0a9f66f4-3203-4f72-bce5-c94005f38c35\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " Nov 03 15:43:44.773296 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.773164 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-workload-socket\") pod \"0a9f66f4-3203-4f72-bce5-c94005f38c35\" (UID: \"0a9f66f4-3203-4f72-bce5-c94005f38c35\") " Nov 03 15:43:44.773871 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.773828 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-credential-socket" pod "0a9f66f4-3203-4f72-bce5-c94005f38c35" (UID: "0a9f66f4-3203-4f72-bce5-c94005f38c35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:44.773871 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.773840 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-workload-certs" pod "0a9f66f4-3203-4f72-bce5-c94005f38c35" (UID: "0a9f66f4-3203-4f72-bce5-c94005f38c35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:44.774487 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.774020 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-data" pod "0a9f66f4-3203-4f72-bce5-c94005f38c35" (UID: "0a9f66f4-3203-4f72-bce5-c94005f38c35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:44.774487 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.774376 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-workload-socket" pod "0a9f66f4-3203-4f72-bce5-c94005f38c35" (UID: "0a9f66f4-3203-4f72-bce5-c94005f38c35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:44.775246 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.775110 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9f66f4-3203-4f72-bce5-c94005f38c35-istiod-ca-cert" pod "0a9f66f4-3203-4f72-bce5-c94005f38c35" (UID: "0a9f66f4-3203-4f72-bce5-c94005f38c35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:43:44.776635 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.776595 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9f66f4-3203-4f72-bce5-c94005f38c35-kube-api-access-8hf7w" pod "0a9f66f4-3203-4f72-bce5-c94005f38c35" (UID: "0a9f66f4-3203-4f72-bce5-c94005f38c35"). InnerVolumeSpecName "kube-api-access-8hf7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:43:44.776763 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.776640 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-token" pod "0a9f66f4-3203-4f72-bce5-c94005f38c35" (UID: "0a9f66f4-3203-4f72-bce5-c94005f38c35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:43:44.776915 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.776886 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-envoy" pod "0a9f66f4-3203-4f72-bce5-c94005f38c35" (UID: "0a9f66f4-3203-4f72-bce5-c94005f38c35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:44.777012 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.776990 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-podinfo" pod "0a9f66f4-3203-4f72-bce5-c94005f38c35" (UID: "0a9f66f4-3203-4f72-bce5-c94005f38c35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:43:44.875049 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.875020 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.875049 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.875045 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f66f4-3203-4f72-bce5-c94005f38c35-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.875049 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.875055 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.875351 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.875064 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.875351 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.875073 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.875351 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.875082 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.875351 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.875090 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9f66f4-3203-4f72-bce5-c94005f38c35-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.875351 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.875098 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hf7w\" (UniqueName: \"kubernetes.io/projected/0a9f66f4-3203-4f72-bce5-c94005f38c35-kube-api-access-8hf7w\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.875351 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.875107 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f66f4-3203-4f72-bce5-c94005f38c35-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.877884 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.877857 4290 generic.go:356] "Generic (PLEG): container finished" podID="0a9f66f4-3203-4f72-bce5-c94005f38c35" containerID="16424b11aec3eec54c63a4117be2d547a01cdc6d4a18fe0d79eb14831674354b" exitCode=0 Nov 03 15:43:44.878029 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.877901 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" Nov 03 15:43:44.878029 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.877939 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" event={"ID":"0a9f66f4-3203-4f72-bce5-c94005f38c35","Type":"ContainerDied","Data":"16424b11aec3eec54c63a4117be2d547a01cdc6d4a18fe0d79eb14831674354b"} Nov 03 15:43:44.878029 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.877987 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g" event={"ID":"0a9f66f4-3203-4f72-bce5-c94005f38c35","Type":"ContainerDied","Data":"87138b4558f80a81cce4d96ff00cfed7d233943e02fc5b0945777e34ec30bf2a"} Nov 03 15:43:44.878029 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.878012 4290 scope.go:122] "RemoveContainer" containerID="16424b11aec3eec54c63a4117be2d547a01cdc6d4a18fe0d79eb14831674354b" Nov 03 15:43:44.881899 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.880934 4290 generic.go:356] "Generic (PLEG): container finished" podID="5a896388-6490-47c0-9eed-5c81fde73165" containerID="c6065e0fedecb88b3e2893d8dcfeabc040d6e02612fa8c72c99332991a862f68" exitCode=0 Nov 03 15:43:44.881899 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.881210 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" Nov 03 15:43:44.881899 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.881902 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" event={"ID":"5a896388-6490-47c0-9eed-5c81fde73165","Type":"ContainerDied","Data":"c6065e0fedecb88b3e2893d8dcfeabc040d6e02612fa8c72c99332991a862f68"} Nov 03 15:43:44.882197 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.881973 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g" event={"ID":"5a896388-6490-47c0-9eed-5c81fde73165","Type":"ContainerDied","Data":"54db47ba24b18f13cc8f2729abb0c5364bda736a279c1ef9547f42f5b50fa382"} Nov 03 15:43:44.897165 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.897143 4290 scope.go:122] "RemoveContainer" containerID="16424b11aec3eec54c63a4117be2d547a01cdc6d4a18fe0d79eb14831674354b" Nov 03 15:43:44.897459 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:43:44.897440 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16424b11aec3eec54c63a4117be2d547a01cdc6d4a18fe0d79eb14831674354b\": container with ID starting with 16424b11aec3eec54c63a4117be2d547a01cdc6d4a18fe0d79eb14831674354b not found: ID does not exist" containerID="16424b11aec3eec54c63a4117be2d547a01cdc6d4a18fe0d79eb14831674354b" Nov 03 15:43:44.897512 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.897471 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16424b11aec3eec54c63a4117be2d547a01cdc6d4a18fe0d79eb14831674354b"} err="failed to get container status \"16424b11aec3eec54c63a4117be2d547a01cdc6d4a18fe0d79eb14831674354b\": rpc error: code = NotFound desc = could not find container \"16424b11aec3eec54c63a4117be2d547a01cdc6d4a18fe0d79eb14831674354b\": container with ID starting with 16424b11aec3eec54c63a4117be2d547a01cdc6d4a18fe0d79eb14831674354b not found: ID does not exist" Nov 03 15:43:44.897512 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.897489 4290 scope.go:122] "RemoveContainer" containerID="c6065e0fedecb88b3e2893d8dcfeabc040d6e02612fa8c72c99332991a862f68" Nov 03 15:43:44.907931 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.907912 4290 scope.go:122] "RemoveContainer" containerID="c6065e0fedecb88b3e2893d8dcfeabc040d6e02612fa8c72c99332991a862f68" Nov 03 15:43:44.908211 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:43:44.908193 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6065e0fedecb88b3e2893d8dcfeabc040d6e02612fa8c72c99332991a862f68\": container with ID starting with c6065e0fedecb88b3e2893d8dcfeabc040d6e02612fa8c72c99332991a862f68 not found: ID does not exist" containerID="c6065e0fedecb88b3e2893d8dcfeabc040d6e02612fa8c72c99332991a862f68" Nov 03 15:43:44.908279 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:44.908225 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6065e0fedecb88b3e2893d8dcfeabc040d6e02612fa8c72c99332991a862f68"} err="failed to get container status \"c6065e0fedecb88b3e2893d8dcfeabc040d6e02612fa8c72c99332991a862f68\": rpc error: code = NotFound desc = could not find container \"c6065e0fedecb88b3e2893d8dcfeabc040d6e02612fa8c72c99332991a862f68\": container with ID starting with c6065e0fedecb88b3e2893d8dcfeabc040d6e02612fa8c72c99332991a862f68 not found: ID does not exist" Nov 03 15:43:45.058813 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:45.058781 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g"] Nov 03 15:43:45.067405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:45.067379 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tds8g"] Nov 03 15:43:45.183068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:45.182984 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g"] Nov 03 15:43:45.187574 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:45.187545 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8nr5g"] Nov 03 15:43:46.680936 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:46.680897 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9f66f4-3203-4f72-bce5-c94005f38c35" path="/var/lib/kubelet/pods/0a9f66f4-3203-4f72-bce5-c94005f38c35/volumes" Nov 03 15:43:46.681424 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:46.681405 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a896388-6490-47c0-9eed-5c81fde73165" path="/var/lib/kubelet/pods/5a896388-6490-47c0-9eed-5c81fde73165/volumes" Nov 03 15:43:51.030343 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.030308 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv"] Nov 03 15:43:51.031028 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.031013 4290 cpu_manager.go:414] "containerMap: removing container" podUID="5a896388-6490-47c0-9eed-5c81fde73165" containerName="istio-proxy" Nov 03 15:43:51.031073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.031030 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a896388-6490-47c0-9eed-5c81fde73165" containerName="istio-proxy" Nov 03 15:43:51.031073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.031042 4290 cpu_manager.go:414] "containerMap: removing container" podUID="0a9f66f4-3203-4f72-bce5-c94005f38c35" containerName="istio-proxy" Nov 03 15:43:51.031073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.031047 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9f66f4-3203-4f72-bce5-c94005f38c35" containerName="istio-proxy" Nov 03 15:43:51.031073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.031055 4290 cpu_manager.go:414] "containerMap: removing container" podUID="6f7d95e8-3d59-481b-bdfd-e33e6854d805" containerName="istio-proxy" Nov 03 15:43:51.031073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.031061 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7d95e8-3d59-481b-bdfd-e33e6854d805" containerName="istio-proxy" Nov 03 15:43:51.031240 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.031088 4290 cpu_manager.go:414] "containerMap: removing container" podUID="764b9a1a-dbcf-4875-8339-ea19b0213e40" containerName="revoke-keys" Nov 03 15:43:51.031240 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.031092 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764b9a1a-dbcf-4875-8339-ea19b0213e40" containerName="revoke-keys" Nov 03 15:43:51.031240 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.031189 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="764b9a1a-dbcf-4875-8339-ea19b0213e40" containerName="revoke-keys" Nov 03 15:43:51.031240 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.031202 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a896388-6490-47c0-9eed-5c81fde73165" containerName="istio-proxy" Nov 03 15:43:51.031240 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.031208 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9f66f4-3203-4f72-bce5-c94005f38c35" containerName="istio-proxy" Nov 03 15:43:51.031240 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.031215 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7d95e8-3d59-481b-bdfd-e33e6854d805" containerName="istio-proxy" Nov 03 15:43:51.036560 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.036538 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:51.059995 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.059955 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-metrics-certs\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:51.060213 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.060012 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69325d2c-8a89-47f0-be5f-f68ff639d398-openshift-service-ca\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:51.060213 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.060101 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-maas-api-tls\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:51.060336 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.060222 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86fhb\" (UniqueName: \"kubernetes.io/projected/69325d2c-8a89-47f0-be5f-f68ff639d398-kube-api-access-86fhb\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:51.092267 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.092237 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5cad072f-serving-cert\"" Nov 03 15:43:51.147534 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.147497 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv"] Nov 03 15:43:51.161681 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.161651 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69325d2c-8a89-47f0-be5f-f68ff639d398-openshift-service-ca\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:51.161821 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.161732 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-maas-api-tls\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:51.161863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.161841 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86fhb\" (UniqueName: \"kubernetes.io/projected/69325d2c-8a89-47f0-be5f-f68ff639d398-kube-api-access-86fhb\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:51.161943 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.161931 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-metrics-certs\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:51.161991 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:43:51.161845 4290 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-serving-cert: secret "maas-api-e2e-mt-5cad072f-serving-cert" not found Nov 03 15:43:51.162071 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:43:51.162057 4290 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-maas-api-tls podName:69325d2c-8a89-47f0-be5f-f68ff639d398 nodeName:}" failed. No retries permitted until 2026-11-03 15:43:51.662034526 +0000 UTC m=+2139.092992314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-maas-api-tls") pod "maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" (UID: "69325d2c-8a89-47f0-be5f-f68ff639d398") : secret "maas-api-e2e-mt-5cad072f-serving-cert" not found Nov 03 15:43:51.162513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.162476 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69325d2c-8a89-47f0-be5f-f68ff639d398-openshift-service-ca\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:51.164526 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.164502 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-metrics-certs\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:51.225154 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.225123 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86fhb\" (UniqueName: \"kubernetes.io/projected/69325d2c-8a89-47f0-be5f-f68ff639d398-kube-api-access-86fhb\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:51.669509 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.669473 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-maas-api-tls\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:51.672339 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.672311 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-maas-api-tls\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:51.952984 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:51.952899 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:52.358763 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:43:52.354352 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69325d2c_8a89_47f0_be5f_f68ff639d398.slice/crio-9960a24ddc301103b3eb574b59245e23131713f68dec891d8ca0a408251580ef WatchSource:0}: Error finding container 9960a24ddc301103b3eb574b59245e23131713f68dec891d8ca0a408251580ef: Status 404 returned error can't find the container with id 9960a24ddc301103b3eb574b59245e23131713f68dec891d8ca0a408251580ef Nov 03 15:43:52.358763 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:52.356541 4290 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:43:52.358763 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:52.357017 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv"] Nov 03 15:43:52.958139 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:52.958104 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" event={"ID":"69325d2c-8a89-47f0-be5f-f68ff639d398","Type":"ContainerStarted","Data":"9960a24ddc301103b3eb574b59245e23131713f68dec891d8ca0a408251580ef"} Nov 03 15:43:53.974352 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:53.974296 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" event={"ID":"69325d2c-8a89-47f0-be5f-f68ff639d398","Type":"ContainerStarted","Data":"fc4e67b6ada6953150351a01a0decd4faf9fb57b5606d53d8055923abb4e268c"} Nov 03 15:43:53.974870 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:53.974402 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:43:54.128793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:54.128731 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" podStartSLOduration=3.256127997 podStartE2EDuration="4.128711355s" totalImagesPullingTime="872.583358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:43:54.126900041 +0000 UTC m=+2141.557857849" watchObservedRunningTime="2026-11-03 15:43:54.128711355 +0000 UTC m=+2141.559669167" Nov 03 15:43:59.989310 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:43:59.989279 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:44:05.639247 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:05.639203 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2"] Nov 03 15:44:05.639688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:05.639432 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" podUID="ce2f1c46-2aae-4048-8008-105f58fceffe" containerName="maas-api" containerID="cri-o://aeee40f2888f9ebb075f9c4c268c4e0e76206a88fbfe4bc49937f5267689c26d" gracePeriod=30 Nov 03 15:44:05.949558 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:05.949528 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:44:06.081761 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.081720 4290 generic.go:356] "Generic (PLEG): container finished" podID="ce2f1c46-2aae-4048-8008-105f58fceffe" containerID="aeee40f2888f9ebb075f9c4c268c4e0e76206a88fbfe4bc49937f5267689c26d" exitCode=0 Nov 03 15:44:06.081952 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.081775 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" event={"ID":"ce2f1c46-2aae-4048-8008-105f58fceffe","Type":"ContainerDied","Data":"aeee40f2888f9ebb075f9c4c268c4e0e76206a88fbfe4bc49937f5267689c26d"} Nov 03 15:44:06.081952 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.081824 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" event={"ID":"ce2f1c46-2aae-4048-8008-105f58fceffe","Type":"ContainerDied","Data":"3112e73167a6d9f5da4313f2181b2a5450b1ae365714e10634ae076bc6bc6a81"} Nov 03 15:44:06.081952 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.081853 4290 scope.go:122] "RemoveContainer" containerID="aeee40f2888f9ebb075f9c4c268c4e0e76206a88fbfe4bc49937f5267689c26d" Nov 03 15:44:06.082102 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.082039 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2" Nov 03 15:44:06.083734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.083688 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce2f1c46-2aae-4048-8008-105f58fceffe-kube-api-access-mqnvz\" (UniqueName: \"kubernetes.io/projected/ce2f1c46-2aae-4048-8008-105f58fceffe-kube-api-access-mqnvz\") pod \"ce2f1c46-2aae-4048-8008-105f58fceffe\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " Nov 03 15:44:06.084613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.084589 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-maas-api-tls\") pod \"ce2f1c46-2aae-4048-8008-105f58fceffe\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " Nov 03 15:44:06.084919 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.084901 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-metrics-certs\") pod \"ce2f1c46-2aae-4048-8008-105f58fceffe\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " Nov 03 15:44:06.085096 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.085077 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce2f1c46-2aae-4048-8008-105f58fceffe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2f1c46-2aae-4048-8008-105f58fceffe-openshift-service-ca\") pod \"ce2f1c46-2aae-4048-8008-105f58fceffe\" (UID: \"ce2f1c46-2aae-4048-8008-105f58fceffe\") " Nov 03 15:44:06.087232 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.087206 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2f1c46-2aae-4048-8008-105f58fceffe-openshift-service-ca" pod "ce2f1c46-2aae-4048-8008-105f58fceffe" (UID: "ce2f1c46-2aae-4048-8008-105f58fceffe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:44:06.088160 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.088129 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-maas-api-tls" pod "ce2f1c46-2aae-4048-8008-105f58fceffe" (UID: "ce2f1c46-2aae-4048-8008-105f58fceffe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:44:06.088880 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.088850 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-metrics-certs" pod "ce2f1c46-2aae-4048-8008-105f58fceffe" (UID: "ce2f1c46-2aae-4048-8008-105f58fceffe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:44:06.089843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.089796 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2f1c46-2aae-4048-8008-105f58fceffe-kube-api-access-mqnvz" pod "ce2f1c46-2aae-4048-8008-105f58fceffe" (UID: "ce2f1c46-2aae-4048-8008-105f58fceffe"). InnerVolumeSpecName "kube-api-access-mqnvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:44:06.123159 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.123132 4290 scope.go:122] "RemoveContainer" containerID="aeee40f2888f9ebb075f9c4c268c4e0e76206a88fbfe4bc49937f5267689c26d" Nov 03 15:44:06.123641 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:44:06.123598 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeee40f2888f9ebb075f9c4c268c4e0e76206a88fbfe4bc49937f5267689c26d\": container with ID starting with aeee40f2888f9ebb075f9c4c268c4e0e76206a88fbfe4bc49937f5267689c26d not found: ID does not exist" containerID="aeee40f2888f9ebb075f9c4c268c4e0e76206a88fbfe4bc49937f5267689c26d" Nov 03 15:44:06.123733 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.123656 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeee40f2888f9ebb075f9c4c268c4e0e76206a88fbfe4bc49937f5267689c26d"} err="failed to get container status \"aeee40f2888f9ebb075f9c4c268c4e0e76206a88fbfe4bc49937f5267689c26d\": rpc error: code = NotFound desc = could not find container \"aeee40f2888f9ebb075f9c4c268c4e0e76206a88fbfe4bc49937f5267689c26d\": container with ID starting with aeee40f2888f9ebb075f9c4c268c4e0e76206a88fbfe4bc49937f5267689c26d not found: ID does not exist" Nov 03 15:44:06.188036 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.187957 4290 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2f1c46-2aae-4048-8008-105f58fceffe-openshift-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:06.188036 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.187982 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqnvz\" (UniqueName: \"kubernetes.io/projected/ce2f1c46-2aae-4048-8008-105f58fceffe-kube-api-access-mqnvz\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:06.188036 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.187996 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-maas-api-tls\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:06.188036 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.188005 4290 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce2f1c46-2aae-4048-8008-105f58fceffe-metrics-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:06.506627 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.506540 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2"] Nov 03 15:44:06.511119 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.511096 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-34e8c5-6b6db67cf4-q9zp2"] Nov 03 15:44:06.699608 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:06.699572 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce2f1c46-2aae-4048-8008-105f58fceffe" path="/var/lib/kubelet/pods/ce2f1c46-2aae-4048-8008-105f58fceffe/volumes" Nov 03 15:44:07.851149 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:07.851120 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t"] Nov 03 15:44:07.904966 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:07.904910 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-34e8c5-jjg6t"] Nov 03 15:44:08.683850 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:08.683759 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="419f8bf7-bc55-41f2-b38d-662f1d834a5c" path="/var/lib/kubelet/pods/419f8bf7-bc55-41f2-b38d-662f1d834a5c/volumes" Nov 03 15:44:08.706820 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:44:08.706779 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:44:15.603560 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:15.603520 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx"] Nov 03 15:44:15.604098 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:15.603788 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" podUID="9d036ddd-5485-4456-986e-a10a2d8c286b" containerName="istio-proxy" containerID="cri-o://733ebefe5736b008c3f8de9ed217c9ed697d7458a2db0fa9d9378c31bcf6b010" gracePeriod=30 Nov 03 15:44:20.052267 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:20.052230 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv"] Nov 03 15:44:20.102126 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:20.102083 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8eea00-ptmpv"] Nov 03 15:44:20.680849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:20.680816 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a52bd2f-9d0b-4782-9e30-da3861709062" path="/var/lib/kubelet/pods/7a52bd2f-9d0b-4782-9e30-da3861709062/volumes" Nov 03 15:44:20.884876 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:20.884852 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:44:21.010413 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.010374 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-token\" (UniqueName: \"kubernetes.io/projected/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-token\") pod \"9d036ddd-5485-4456-986e-a10a2d8c286b\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " Nov 03 15:44:21.010571 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.010534 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-envoy\") pod \"9d036ddd-5485-4456-986e-a10a2d8c286b\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " Nov 03 15:44:21.010571 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.010566 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-workload-certs\") pod \"9d036ddd-5485-4456-986e-a10a2d8c286b\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " Nov 03 15:44:21.010650 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.010607 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d036ddd-5485-4456-986e-a10a2d8c286b-kube-api-access-nzc7q\" (UniqueName: \"kubernetes.io/projected/9d036ddd-5485-4456-986e-a10a2d8c286b-kube-api-access-nzc7q\") pod \"9d036ddd-5485-4456-986e-a10a2d8c286b\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " Nov 03 15:44:21.010650 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.010640 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d036ddd-5485-4456-986e-a10a2d8c286b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d036ddd-5485-4456-986e-a10a2d8c286b-istiod-ca-cert\") pod \"9d036ddd-5485-4456-986e-a10a2d8c286b\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " Nov 03 15:44:21.010781 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.010679 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-data\") pod \"9d036ddd-5485-4456-986e-a10a2d8c286b\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " Nov 03 15:44:21.010781 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.010760 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-podinfo\") pod \"9d036ddd-5485-4456-986e-a10a2d8c286b\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " Nov 03 15:44:21.010907 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.010798 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-workload-socket\") pod \"9d036ddd-5485-4456-986e-a10a2d8c286b\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " Nov 03 15:44:21.010965 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.010933 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-credential-socket\") pod \"9d036ddd-5485-4456-986e-a10a2d8c286b\" (UID: \"9d036ddd-5485-4456-986e-a10a2d8c286b\") " Nov 03 15:44:21.011032 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.010937 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-workload-certs" pod "9d036ddd-5485-4456-986e-a10a2d8c286b" (UID: "9d036ddd-5485-4456-986e-a10a2d8c286b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:44:21.011208 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.011101 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d036ddd-5485-4456-986e-a10a2d8c286b-istiod-ca-cert" pod "9d036ddd-5485-4456-986e-a10a2d8c286b" (UID: "9d036ddd-5485-4456-986e-a10a2d8c286b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:44:21.011460 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.011413 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-data" pod "9d036ddd-5485-4456-986e-a10a2d8c286b" (UID: "9d036ddd-5485-4456-986e-a10a2d8c286b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:44:21.011658 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.011605 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-credential-socket" pod "9d036ddd-5485-4456-986e-a10a2d8c286b" (UID: "9d036ddd-5485-4456-986e-a10a2d8c286b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:44:21.011845 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.011816 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-workload-socket" pod "9d036ddd-5485-4456-986e-a10a2d8c286b" (UID: "9d036ddd-5485-4456-986e-a10a2d8c286b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:44:21.011949 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.011902 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:21.011949 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.011924 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d036ddd-5485-4456-986e-a10a2d8c286b-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:21.011949 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.011939 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:21.012118 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.011954 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:21.012934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.012904 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-envoy" pod "9d036ddd-5485-4456-986e-a10a2d8c286b" (UID: "9d036ddd-5485-4456-986e-a10a2d8c286b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:44:21.013032 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.012977 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-podinfo" pod "9d036ddd-5485-4456-986e-a10a2d8c286b" (UID: "9d036ddd-5485-4456-986e-a10a2d8c286b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:44:21.013121 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.013105 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-token" pod "9d036ddd-5485-4456-986e-a10a2d8c286b" (UID: "9d036ddd-5485-4456-986e-a10a2d8c286b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:44:21.013167 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.013136 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d036ddd-5485-4456-986e-a10a2d8c286b-kube-api-access-nzc7q" pod "9d036ddd-5485-4456-986e-a10a2d8c286b" (UID: "9d036ddd-5485-4456-986e-a10a2d8c286b"). InnerVolumeSpecName "kube-api-access-nzc7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:44:21.113657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.113623 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:21.113657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.113653 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzc7q\" (UniqueName: \"kubernetes.io/projected/9d036ddd-5485-4456-986e-a10a2d8c286b-kube-api-access-nzc7q\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:21.114135 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.113667 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:21.114135 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.113676 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d036ddd-5485-4456-986e-a10a2d8c286b-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:21.114135 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.113692 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d036ddd-5485-4456-986e-a10a2d8c286b-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:21.237378 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.237282 4290 generic.go:356] "Generic (PLEG): container finished" podID="9d036ddd-5485-4456-986e-a10a2d8c286b" containerID="733ebefe5736b008c3f8de9ed217c9ed697d7458a2db0fa9d9378c31bcf6b010" exitCode=0 Nov 03 15:44:21.237551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.237420 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" event={"ID":"9d036ddd-5485-4456-986e-a10a2d8c286b","Type":"ContainerDied","Data":"733ebefe5736b008c3f8de9ed217c9ed697d7458a2db0fa9d9378c31bcf6b010"} Nov 03 15:44:21.237551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.237431 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" Nov 03 15:44:21.237551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.237507 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx" event={"ID":"9d036ddd-5485-4456-986e-a10a2d8c286b","Type":"ContainerDied","Data":"e86a829bdc7e50a4a9e0ba60a4808f407cd0ed14f396a8a39de3d412e794ecc4"} Nov 03 15:44:21.237551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.237536 4290 scope.go:122] "RemoveContainer" containerID="733ebefe5736b008c3f8de9ed217c9ed697d7458a2db0fa9d9378c31bcf6b010" Nov 03 15:44:21.253026 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.252998 4290 scope.go:122] "RemoveContainer" containerID="733ebefe5736b008c3f8de9ed217c9ed697d7458a2db0fa9d9378c31bcf6b010" Nov 03 15:44:21.256637 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:44:21.256091 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733ebefe5736b008c3f8de9ed217c9ed697d7458a2db0fa9d9378c31bcf6b010\": container with ID starting with 733ebefe5736b008c3f8de9ed217c9ed697d7458a2db0fa9d9378c31bcf6b010 not found: ID does not exist" containerID="733ebefe5736b008c3f8de9ed217c9ed697d7458a2db0fa9d9378c31bcf6b010" Nov 03 15:44:21.256637 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.256130 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733ebefe5736b008c3f8de9ed217c9ed697d7458a2db0fa9d9378c31bcf6b010"} err="failed to get container status \"733ebefe5736b008c3f8de9ed217c9ed697d7458a2db0fa9d9378c31bcf6b010\": rpc error: code = NotFound desc = could not find container \"733ebefe5736b008c3f8de9ed217c9ed697d7458a2db0fa9d9378c31bcf6b010\": container with ID starting with 733ebefe5736b008c3f8de9ed217c9ed697d7458a2db0fa9d9378c31bcf6b010 not found: ID does not exist" Nov 03 15:44:21.407289 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.407252 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx"] Nov 03 15:44:21.412662 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:21.412637 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xg5nx"] Nov 03 15:44:22.680816 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:22.680788 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d036ddd-5485-4456-986e-a10a2d8c286b" path="/var/lib/kubelet/pods/9d036ddd-5485-4456-986e-a10a2d8c286b/volumes" Nov 03 15:44:24.909893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:24.909859 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6"] Nov 03 15:44:24.959276 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:24.959246 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-771ea0-p6hw6"] Nov 03 15:44:25.311541 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.311457 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn"] Nov 03 15:44:25.312287 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.312264 4290 cpu_manager.go:414] "containerMap: removing container" podUID="9d036ddd-5485-4456-986e-a10a2d8c286b" containerName="istio-proxy" Nov 03 15:44:25.312287 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.312286 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d036ddd-5485-4456-986e-a10a2d8c286b" containerName="istio-proxy" Nov 03 15:44:25.312487 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.312341 4290 cpu_manager.go:414] "containerMap: removing container" podUID="ce2f1c46-2aae-4048-8008-105f58fceffe" containerName="maas-api" Nov 03 15:44:25.312487 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.312349 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2f1c46-2aae-4048-8008-105f58fceffe" containerName="maas-api" Nov 03 15:44:25.312593 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.312502 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2f1c46-2aae-4048-8008-105f58fceffe" containerName="maas-api" Nov 03 15:44:25.312593 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.312525 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d036ddd-5485-4456-986e-a10a2d8c286b" containerName="istio-proxy" Nov 03 15:44:25.317421 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.317399 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" Nov 03 15:44:25.373969 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.373941 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn"] Nov 03 15:44:25.450799 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.450768 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:44:25.450799 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.450790 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:44:25.476811 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.476776 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjtmn\" (UniqueName: \"kubernetes.io/projected/0ca101a0-06bb-4911-997f-be22cb2a1574-kube-api-access-sjtmn\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn\" (UID: \"0ca101a0-06bb-4911-997f-be22cb2a1574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" Nov 03 15:44:25.476997 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.476968 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca101a0-06bb-4911-997f-be22cb2a1574-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn\" (UID: \"0ca101a0-06bb-4911-997f-be22cb2a1574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" Nov 03 15:44:25.477062 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.477039 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca101a0-06bb-4911-997f-be22cb2a1574-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn\" (UID: \"0ca101a0-06bb-4911-997f-be22cb2a1574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" Nov 03 15:44:25.578162 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.578085 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca101a0-06bb-4911-997f-be22cb2a1574-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn\" (UID: \"0ca101a0-06bb-4911-997f-be22cb2a1574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" Nov 03 15:44:25.578313 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.578200 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjtmn\" (UniqueName: \"kubernetes.io/projected/0ca101a0-06bb-4911-997f-be22cb2a1574-kube-api-access-sjtmn\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn\" (UID: \"0ca101a0-06bb-4911-997f-be22cb2a1574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" Nov 03 15:44:25.578313 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.578299 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca101a0-06bb-4911-997f-be22cb2a1574-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn\" (UID: \"0ca101a0-06bb-4911-997f-be22cb2a1574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" Nov 03 15:44:25.578995 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.578931 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca101a0-06bb-4911-997f-be22cb2a1574-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn\" (UID: \"0ca101a0-06bb-4911-997f-be22cb2a1574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" Nov 03 15:44:25.580607 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.580576 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca101a0-06bb-4911-997f-be22cb2a1574-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn\" (UID: \"0ca101a0-06bb-4911-997f-be22cb2a1574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" Nov 03 15:44:25.640432 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.640405 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjtmn\" (UniqueName: \"kubernetes.io/projected/0ca101a0-06bb-4911-997f-be22cb2a1574-kube-api-access-sjtmn\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn\" (UID: \"0ca101a0-06bb-4911-997f-be22cb2a1574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" Nov 03 15:44:25.931608 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:25.931580 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" Nov 03 15:44:26.335259 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:44:26.335231 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca101a0_06bb_4911_997f_be22cb2a1574.slice/crio-aa22d09d0cef1d3abd24be8b0ed5cb7caa7f60cc4feea93f1eadae9febaf81a0 WatchSource:0}: Error finding container aa22d09d0cef1d3abd24be8b0ed5cb7caa7f60cc4feea93f1eadae9febaf81a0: Status 404 returned error can't find the container with id aa22d09d0cef1d3abd24be8b0ed5cb7caa7f60cc4feea93f1eadae9febaf81a0 Nov 03 15:44:26.336487 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:26.336460 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn"] Nov 03 15:44:26.683331 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:26.683299 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="764b9a1a-dbcf-4875-8339-ea19b0213e40" path="/var/lib/kubelet/pods/764b9a1a-dbcf-4875-8339-ea19b0213e40/volumes" Nov 03 15:44:27.292680 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:27.292642 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" event={"ID":"0ca101a0-06bb-4911-997f-be22cb2a1574","Type":"ContainerStarted","Data":"21c7f14d2cb2ad8161617d47ed6b3277c4b32514bd309852fc7598ecbb63927c"} Nov 03 15:44:27.293104 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:27.292718 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" event={"ID":"0ca101a0-06bb-4911-997f-be22cb2a1574","Type":"ContainerStarted","Data":"aa22d09d0cef1d3abd24be8b0ed5cb7caa7f60cc4feea93f1eadae9febaf81a0"} Nov 03 15:44:27.414128 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:27.414070 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" podStartSLOduration=2.414051727 podStartE2EDuration="2.414051727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:44:27.408480541 +0000 UTC m=+2174.839438368" watchObservedRunningTime="2026-11-03 15:44:27.414051727 +0000 UTC m=+2174.845009534" Nov 03 15:44:28.303048 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:28.303016 4290 generic.go:356] "Generic (PLEG): container finished" podID="0ca101a0-06bb-4911-997f-be22cb2a1574" containerID="21c7f14d2cb2ad8161617d47ed6b3277c4b32514bd309852fc7598ecbb63927c" exitCode=0 Nov 03 15:44:28.303476 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:28.303112 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" event={"ID":"0ca101a0-06bb-4911-997f-be22cb2a1574","Type":"ContainerDied","Data":"21c7f14d2cb2ad8161617d47ed6b3277c4b32514bd309852fc7598ecbb63927c"} Nov 03 15:44:29.489388 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:29.489357 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" Nov 03 15:44:29.632134 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:29.632104 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca101a0-06bb-4911-997f-be22cb2a1574-kube-api-access-sjtmn\" (UniqueName: \"kubernetes.io/projected/0ca101a0-06bb-4911-997f-be22cb2a1574-kube-api-access-sjtmn\") pod \"0ca101a0-06bb-4911-997f-be22cb2a1574\" (UID: \"0ca101a0-06bb-4911-997f-be22cb2a1574\") " Nov 03 15:44:29.632320 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:29.632152 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ca101a0-06bb-4911-997f-be22cb2a1574-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca101a0-06bb-4911-997f-be22cb2a1574-maas-api-openshift-ca\") pod \"0ca101a0-06bb-4911-997f-be22cb2a1574\" (UID: \"0ca101a0-06bb-4911-997f-be22cb2a1574\") " Nov 03 15:44:29.632320 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:29.632204 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ca101a0-06bb-4911-997f-be22cb2a1574-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca101a0-06bb-4911-997f-be22cb2a1574-maas-api-xks-ca\") pod \"0ca101a0-06bb-4911-997f-be22cb2a1574\" (UID: \"0ca101a0-06bb-4911-997f-be22cb2a1574\") " Nov 03 15:44:29.632630 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:29.632598 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca101a0-06bb-4911-997f-be22cb2a1574-maas-api-openshift-ca" pod "0ca101a0-06bb-4911-997f-be22cb2a1574" (UID: "0ca101a0-06bb-4911-997f-be22cb2a1574"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:44:29.632799 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:29.632767 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca101a0-06bb-4911-997f-be22cb2a1574-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:29.635024 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:29.635001 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca101a0-06bb-4911-997f-be22cb2a1574-maas-api-xks-ca" pod "0ca101a0-06bb-4911-997f-be22cb2a1574" (UID: "0ca101a0-06bb-4911-997f-be22cb2a1574"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:44:29.635024 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:29.635009 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca101a0-06bb-4911-997f-be22cb2a1574-kube-api-access-sjtmn" pod "0ca101a0-06bb-4911-997f-be22cb2a1574" (UID: "0ca101a0-06bb-4911-997f-be22cb2a1574"). InnerVolumeSpecName "kube-api-access-sjtmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:44:29.733913 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:29.733879 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca101a0-06bb-4911-997f-be22cb2a1574-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:29.733913 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:29.733908 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjtmn\" (UniqueName: \"kubernetes.io/projected/0ca101a0-06bb-4911-997f-be22cb2a1574-kube-api-access-sjtmn\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:30.321578 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:30.321537 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" event={"ID":"0ca101a0-06bb-4911-997f-be22cb2a1574","Type":"ContainerDied","Data":"aa22d09d0cef1d3abd24be8b0ed5cb7caa7f60cc4feea93f1eadae9febaf81a0"} Nov 03 15:44:30.321816 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:30.321594 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa22d09d0cef1d3abd24be8b0ed5cb7caa7f60cc4feea93f1eadae9febaf81a0" Nov 03 15:44:30.321816 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:30.321553 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn" Nov 03 15:44:33.104386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.104338 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5"] Nov 03 15:44:33.105401 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.105381 4290 cpu_manager.go:414] "containerMap: removing container" podUID="0ca101a0-06bb-4911-997f-be22cb2a1574" containerName="revoke-keys" Nov 03 15:44:33.105401 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.105401 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca101a0-06bb-4911-997f-be22cb2a1574" containerName="revoke-keys" Nov 03 15:44:33.105543 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.105536 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca101a0-06bb-4911-997f-be22cb2a1574" containerName="revoke-keys" Nov 03 15:44:33.111017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.110991 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" Nov 03 15:44:33.167996 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.167966 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:44:33.168237 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.168071 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5"] Nov 03 15:44:33.168396 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.168149 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:44:33.194109 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.194078 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babcf474-65c1-4912-b599-ca9c31e24276-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5\" (UID: \"babcf474-65c1-4912-b599-ca9c31e24276\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" Nov 03 15:44:33.194293 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.194162 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babcf474-65c1-4912-b599-ca9c31e24276-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5\" (UID: \"babcf474-65c1-4912-b599-ca9c31e24276\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" Nov 03 15:44:33.194293 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.194256 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h48kr\" (UniqueName: \"kubernetes.io/projected/babcf474-65c1-4912-b599-ca9c31e24276-kube-api-access-h48kr\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5\" (UID: \"babcf474-65c1-4912-b599-ca9c31e24276\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" Nov 03 15:44:33.295613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.295580 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babcf474-65c1-4912-b599-ca9c31e24276-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5\" (UID: \"babcf474-65c1-4912-b599-ca9c31e24276\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" Nov 03 15:44:33.295613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.295615 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h48kr\" (UniqueName: \"kubernetes.io/projected/babcf474-65c1-4912-b599-ca9c31e24276-kube-api-access-h48kr\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5\" (UID: \"babcf474-65c1-4912-b599-ca9c31e24276\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" Nov 03 15:44:33.295869 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.295838 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babcf474-65c1-4912-b599-ca9c31e24276-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5\" (UID: \"babcf474-65c1-4912-b599-ca9c31e24276\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" Nov 03 15:44:33.296289 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.296264 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babcf474-65c1-4912-b599-ca9c31e24276-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5\" (UID: \"babcf474-65c1-4912-b599-ca9c31e24276\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" Nov 03 15:44:33.298309 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.298285 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babcf474-65c1-4912-b599-ca9c31e24276-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5\" (UID: \"babcf474-65c1-4912-b599-ca9c31e24276\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" Nov 03 15:44:33.357567 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.357494 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h48kr\" (UniqueName: \"kubernetes.io/projected/babcf474-65c1-4912-b599-ca9c31e24276-kube-api-access-h48kr\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5\" (UID: \"babcf474-65c1-4912-b599-ca9c31e24276\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" Nov 03 15:44:33.427388 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.427360 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" Nov 03 15:44:33.818444 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:33.818408 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5"] Nov 03 15:44:33.827580 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:44:33.827547 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabcf474_65c1_4912_b599_ca9c31e24276.slice/crio-6d52a3f4de56df7839fb72fdd550958ba110e370408cb6ebba3469fd579bc66f WatchSource:0}: Error finding container 6d52a3f4de56df7839fb72fdd550958ba110e370408cb6ebba3469fd579bc66f: Status 404 returned error can't find the container with id 6d52a3f4de56df7839fb72fdd550958ba110e370408cb6ebba3469fd579bc66f Nov 03 15:44:34.361773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:34.361733 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" event={"ID":"babcf474-65c1-4912-b599-ca9c31e24276","Type":"ContainerStarted","Data":"43053fdafdebe78a27760834c5a65cabe96eadd852487e6236f1590049cd3557"} Nov 03 15:44:34.362195 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:34.361802 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" event={"ID":"babcf474-65c1-4912-b599-ca9c31e24276","Type":"ContainerStarted","Data":"6d52a3f4de56df7839fb72fdd550958ba110e370408cb6ebba3469fd579bc66f"} Nov 03 15:44:34.498337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:34.498235 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" podStartSLOduration=1.498221891 podStartE2EDuration="1.498221891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:44:34.494268113 +0000 UTC m=+2181.925225920" watchObservedRunningTime="2026-11-03 15:44:34.498221891 +0000 UTC m=+2181.929179697" Nov 03 15:44:35.374634 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.374602 4290 generic.go:356] "Generic (PLEG): container finished" podID="babcf474-65c1-4912-b599-ca9c31e24276" containerID="43053fdafdebe78a27760834c5a65cabe96eadd852487e6236f1590049cd3557" exitCode=0 Nov 03 15:44:35.375065 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.374688 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" event={"ID":"babcf474-65c1-4912-b599-ca9c31e24276","Type":"ContainerDied","Data":"43053fdafdebe78a27760834c5a65cabe96eadd852487e6236f1590049cd3557"} Nov 03 15:44:35.779316 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.779233 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng"] Nov 03 15:44:35.784919 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.784893 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.828361 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.828328 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt"] Nov 03 15:44:35.832694 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.832678 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:35.832864 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.832844 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.832907 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.832882 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.833023 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.833000 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.833082 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.833068 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.833199 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.833183 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.833258 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.833246 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.833360 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.833343 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhvd2\" (UniqueName: \"kubernetes.io/projected/3f80f0c3-6c46-4763-bd3c-0b4737b40044-kube-api-access-dhvd2\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.833432 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.833366 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.833432 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.833395 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.842978 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.842954 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5cad072f-openshift-default-dockercfg-2j2g5\"" Nov 03 15:44:35.845756 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.845691 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng"] Nov 03 15:44:35.894960 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.894921 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt"] Nov 03 15:44:35.935050 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935010 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04ac745-e80e-433c-a131-d61612494260-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:35.935222 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935122 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04ac745-e80e-433c-a131-d61612494260-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:35.935222 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935174 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04ac745-e80e-433c-a131-d61612494260-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:35.935222 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935196 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:35.935222 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935215 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.935450 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935430 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.935508 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935467 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:35.935508 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935485 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:35.935624 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935526 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:35.935624 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935605 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhvd2\" (UniqueName: \"kubernetes.io/projected/3f80f0c3-6c46-4763-bd3c-0b4737b40044-kube-api-access-dhvd2\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.935780 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935633 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.935780 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935652 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:35.935780 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935696 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.935968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935791 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.935968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935856 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.935968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935895 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.935968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935926 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.935968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.935940 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2fmz\" (UniqueName: \"kubernetes.io/projected/d04ac745-e80e-433c-a131-d61612494260-kube-api-access-d2fmz\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:35.936538 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.936073 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.936538 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.936152 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.936538 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.936214 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.936538 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.936446 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.936671 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.936582 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.937587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.937561 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.938521 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.938500 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.998048 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.998012 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:35.998214 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:35.998059 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhvd2\" (UniqueName: \"kubernetes.io/projected/3f80f0c3-6c46-4763-bd3c-0b4737b40044-kube-api-access-dhvd2\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:36.037257 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.037189 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.037397 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.037278 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.037397 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.037311 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.037478 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.037437 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.037533 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.037484 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.037591 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.037560 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.037642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.037599 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.037642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.037625 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2fmz\" (UniqueName: \"kubernetes.io/projected/d04ac745-e80e-433c-a131-d61612494260-kube-api-access-d2fmz\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.037840 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.037669 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.037840 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.037689 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04ac745-e80e-433c-a131-d61612494260-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.037840 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.037811 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04ac745-e80e-433c-a131-d61612494260-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.037972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.037839 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.037972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.037854 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04ac745-e80e-433c-a131-d61612494260-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.038263 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.038244 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04ac745-e80e-433c-a131-d61612494260-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.040245 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.040222 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.040336 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.040257 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04ac745-e80e-433c-a131-d61612494260-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.100234 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.100208 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04ac745-e80e-433c-a131-d61612494260-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.100399 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.100270 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2fmz\" (UniqueName: \"kubernetes.io/projected/d04ac745-e80e-433c-a131-d61612494260-kube-api-access-d2fmz\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.102155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.102138 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:36.150737 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.150689 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:36.538155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.538124 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" Nov 03 15:44:36.542544 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.542422 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng"] Nov 03 15:44:36.549931 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.549886 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:44:36.550020 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.549986 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:44:36.550055 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.550034 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:44:36.600461 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:44:36.600430 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04ac745_e80e_433c_a131_d61612494260.slice/crio-d0ddf178a1c9b4647c6c47efe94d0e9b09a195db51309f4e6dd1d0c6d2918559 WatchSource:0}: Error finding container d0ddf178a1c9b4647c6c47efe94d0e9b09a195db51309f4e6dd1d0c6d2918559: Status 404 returned error can't find the container with id d0ddf178a1c9b4647c6c47efe94d0e9b09a195db51309f4e6dd1d0c6d2918559 Nov 03 15:44:36.600758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.600736 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt"] Nov 03 15:44:36.604046 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.604004 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:44:36.604138 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.604100 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:44:36.604213 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.604142 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:44:36.649051 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.649020 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/babcf474-65c1-4912-b599-ca9c31e24276-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babcf474-65c1-4912-b599-ca9c31e24276-maas-api-xks-ca\") pod \"babcf474-65c1-4912-b599-ca9c31e24276\" (UID: \"babcf474-65c1-4912-b599-ca9c31e24276\") " Nov 03 15:44:36.649171 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.649146 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/babcf474-65c1-4912-b599-ca9c31e24276-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babcf474-65c1-4912-b599-ca9c31e24276-maas-api-openshift-ca\") pod \"babcf474-65c1-4912-b599-ca9c31e24276\" (UID: \"babcf474-65c1-4912-b599-ca9c31e24276\") " Nov 03 15:44:36.649331 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.649300 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/babcf474-65c1-4912-b599-ca9c31e24276-kube-api-access-h48kr\" (UniqueName: \"kubernetes.io/projected/babcf474-65c1-4912-b599-ca9c31e24276-kube-api-access-h48kr\") pod \"babcf474-65c1-4912-b599-ca9c31e24276\" (UID: \"babcf474-65c1-4912-b599-ca9c31e24276\") " Nov 03 15:44:36.649512 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.649488 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babcf474-65c1-4912-b599-ca9c31e24276-maas-api-openshift-ca" pod "babcf474-65c1-4912-b599-ca9c31e24276" (UID: "babcf474-65c1-4912-b599-ca9c31e24276"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:44:36.649971 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.649948 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babcf474-65c1-4912-b599-ca9c31e24276-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:36.652045 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.652018 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babcf474-65c1-4912-b599-ca9c31e24276-kube-api-access-h48kr" pod "babcf474-65c1-4912-b599-ca9c31e24276" (UID: "babcf474-65c1-4912-b599-ca9c31e24276"). InnerVolumeSpecName "kube-api-access-h48kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:44:36.652148 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.652120 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/babcf474-65c1-4912-b599-ca9c31e24276-maas-api-xks-ca" pod "babcf474-65c1-4912-b599-ca9c31e24276" (UID: "babcf474-65c1-4912-b599-ca9c31e24276"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:44:36.754253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.752113 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babcf474-65c1-4912-b599-ca9c31e24276-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:36.754253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:36.752150 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h48kr\" (UniqueName: \"kubernetes.io/projected/babcf474-65c1-4912-b599-ca9c31e24276-kube-api-access-h48kr\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:37.436998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:37.436954 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" Nov 03 15:44:37.436998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:37.436967 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5" event={"ID":"babcf474-65c1-4912-b599-ca9c31e24276","Type":"ContainerDied","Data":"6d52a3f4de56df7839fb72fdd550958ba110e370408cb6ebba3469fd579bc66f"} Nov 03 15:44:37.437266 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:37.437023 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d52a3f4de56df7839fb72fdd550958ba110e370408cb6ebba3469fd579bc66f" Nov 03 15:44:37.465743 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:37.462134 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" event={"ID":"d04ac745-e80e-433c-a131-d61612494260","Type":"ContainerStarted","Data":"13449d28964d81dab54958cb8af42ff6a7f58e2ec5ccdc85839c30cc4ec26602"} Nov 03 15:44:37.465743 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:37.462217 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" event={"ID":"d04ac745-e80e-433c-a131-d61612494260","Type":"ContainerStarted","Data":"d0ddf178a1c9b4647c6c47efe94d0e9b09a195db51309f4e6dd1d0c6d2918559"} Nov 03 15:44:37.486522 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:37.486484 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" event={"ID":"3f80f0c3-6c46-4763-bd3c-0b4737b40044","Type":"ContainerStarted","Data":"b1ed34700915c2e076c5d7c900d01a4ba73e87e5bc3ce4c08190d25d75835e42"} Nov 03 15:44:37.486850 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:37.486800 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" event={"ID":"3f80f0c3-6c46-4763-bd3c-0b4737b40044","Type":"ContainerStarted","Data":"0958850f6b2c4c27e0a9343221c4f67fe16b58829a702f53f7f16cff5c605fe8"} Nov 03 15:44:37.591424 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:37.591349 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" podStartSLOduration=2.59132832 podStartE2EDuration="2.59132832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:44:37.586874769 +0000 UTC m=+2185.017832576" watchObservedRunningTime="2026-11-03 15:44:37.59132832 +0000 UTC m=+2185.022286126" Nov 03 15:44:37.796805 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:37.796661 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" podStartSLOduration=2.796637503 podStartE2EDuration="2.796637503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:44:37.765284496 +0000 UTC m=+2185.196242305" watchObservedRunningTime="2026-11-03 15:44:37.796637503 +0000 UTC m=+2185.227595310" Nov 03 15:44:38.103327 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:38.103237 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:38.104927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:38.104899 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 15:44:38.105057 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:38.104956 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" podUID="3f80f0c3-6c46-4763-bd3c-0b4737b40044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 15:44:38.166729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:38.155913 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:38.166729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:38.156073 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 15:44:38.166729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:38.156119 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" podUID="d04ac745-e80e-433c-a131-d61612494260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 15:44:38.821180 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:44:38.821143 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:44:39.102731 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:39.102610 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 15:44:39.102927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:39.102690 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" podUID="3f80f0c3-6c46-4763-bd3c-0b4737b40044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 15:44:39.157744 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:39.151236 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 15:44:39.157744 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:39.151316 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" podUID="d04ac745-e80e-433c-a131-d61612494260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 15:44:40.103513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.103472 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 15:44:40.104043 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.103537 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" podUID="3f80f0c3-6c46-4763-bd3c-0b4737b40044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 15:44:40.157744 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.154174 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 15:44:40.157744 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.154242 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" podUID="d04ac745-e80e-433c-a131-d61612494260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 15:44:40.799781 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.799355 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4"] Nov 03 15:44:40.800373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.800343 4290 cpu_manager.go:414] "containerMap: removing container" podUID="babcf474-65c1-4912-b599-ca9c31e24276" containerName="revoke-keys" Nov 03 15:44:40.800373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.800366 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="babcf474-65c1-4912-b599-ca9c31e24276" containerName="revoke-keys" Nov 03 15:44:40.800613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.800597 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="babcf474-65c1-4912-b599-ca9c31e24276" containerName="revoke-keys" Nov 03 15:44:40.804211 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.804191 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" Nov 03 15:44:40.834576 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.834547 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8115d771-05bd-4b84-bc55-2817cf4b2fa4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4\" (UID: \"8115d771-05bd-4b84-bc55-2817cf4b2fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" Nov 03 15:44:40.834681 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.834640 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8115d771-05bd-4b84-bc55-2817cf4b2fa4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4\" (UID: \"8115d771-05bd-4b84-bc55-2817cf4b2fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" Nov 03 15:44:40.834847 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.834811 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v7s6\" (UniqueName: \"kubernetes.io/projected/8115d771-05bd-4b84-bc55-2817cf4b2fa4-kube-api-access-9v7s6\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4\" (UID: \"8115d771-05bd-4b84-bc55-2817cf4b2fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" Nov 03 15:44:40.899118 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.899084 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:44:40.899384 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.899367 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:44:40.901252 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.901217 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4"] Nov 03 15:44:40.937528 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.936340 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v7s6\" (UniqueName: \"kubernetes.io/projected/8115d771-05bd-4b84-bc55-2817cf4b2fa4-kube-api-access-9v7s6\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4\" (UID: \"8115d771-05bd-4b84-bc55-2817cf4b2fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" Nov 03 15:44:40.937528 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.936434 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8115d771-05bd-4b84-bc55-2817cf4b2fa4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4\" (UID: \"8115d771-05bd-4b84-bc55-2817cf4b2fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" Nov 03 15:44:40.937528 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.936507 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8115d771-05bd-4b84-bc55-2817cf4b2fa4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4\" (UID: \"8115d771-05bd-4b84-bc55-2817cf4b2fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" Nov 03 15:44:40.937528 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.937215 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8115d771-05bd-4b84-bc55-2817cf4b2fa4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4\" (UID: \"8115d771-05bd-4b84-bc55-2817cf4b2fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" Nov 03 15:44:40.993787 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:40.991542 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8115d771-05bd-4b84-bc55-2817cf4b2fa4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4\" (UID: \"8115d771-05bd-4b84-bc55-2817cf4b2fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" Nov 03 15:44:41.000962 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:41.000501 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v7s6\" (UniqueName: \"kubernetes.io/projected/8115d771-05bd-4b84-bc55-2817cf4b2fa4-kube-api-access-9v7s6\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4\" (UID: \"8115d771-05bd-4b84-bc55-2817cf4b2fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" Nov 03 15:44:41.105216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:41.103948 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 15:44:41.105216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:41.104018 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" podUID="3f80f0c3-6c46-4763-bd3c-0b4737b40044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 15:44:41.118353 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:41.118320 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" Nov 03 15:44:41.167727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:41.165613 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 15:44:41.167727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:41.165668 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" podUID="d04ac745-e80e-433c-a131-d61612494260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 15:44:41.608977 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:41.608885 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4"] Nov 03 15:44:41.610901 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:44:41.610872 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8115d771_05bd_4b84_bc55_2817cf4b2fa4.slice/crio-7844b83affcc22246f624951c1134457e98f2ec32dd7164a4fcba65ce4d63876 WatchSource:0}: Error finding container 7844b83affcc22246f624951c1134457e98f2ec32dd7164a4fcba65ce4d63876: Status 404 returned error can't find the container with id 7844b83affcc22246f624951c1134457e98f2ec32dd7164a4fcba65ce4d63876 Nov 03 15:44:42.106758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:42.106684 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 15:44:42.107288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:42.106794 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" podUID="3f80f0c3-6c46-4763-bd3c-0b4737b40044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 15:44:42.152557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:42.152442 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 15:44:42.152557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:42.152502 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" podUID="d04ac745-e80e-433c-a131-d61612494260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 15:44:42.618807 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:42.618719 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" event={"ID":"8115d771-05bd-4b84-bc55-2817cf4b2fa4","Type":"ContainerStarted","Data":"daaa9d3712c33845c3f7ccd1b2a9de00bd2ff68248005d78b72ad6ea4f81055e"} Nov 03 15:44:42.618807 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:42.618775 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" event={"ID":"8115d771-05bd-4b84-bc55-2817cf4b2fa4","Type":"ContainerStarted","Data":"7844b83affcc22246f624951c1134457e98f2ec32dd7164a4fcba65ce4d63876"} Nov 03 15:44:42.755772 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:42.755686 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" podStartSLOduration=2.755667341 podStartE2EDuration="2.755667341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:44:42.75093452 +0000 UTC m=+2190.181892339" watchObservedRunningTime="2026-11-03 15:44:42.755667341 +0000 UTC m=+2190.186625149" Nov 03 15:44:43.332668 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:43.332634 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:43.333647 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:43.333621 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:43.335473 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:43.335454 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:44:43.388744 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:43.386884 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:43.388744 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:43.388280 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:43.389243 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:43.389145 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:44:43.634533 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:43.634497 4290 generic.go:356] "Generic (PLEG): container finished" podID="8115d771-05bd-4b84-bc55-2817cf4b2fa4" containerID="daaa9d3712c33845c3f7ccd1b2a9de00bd2ff68248005d78b72ad6ea4f81055e" exitCode=0 Nov 03 15:44:43.635262 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:43.634596 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" event={"ID":"8115d771-05bd-4b84-bc55-2817cf4b2fa4","Type":"ContainerDied","Data":"daaa9d3712c33845c3f7ccd1b2a9de00bd2ff68248005d78b72ad6ea4f81055e"} Nov 03 15:44:44.826515 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:44.826488 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" Nov 03 15:44:44.953791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:44.953675 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8115d771-05bd-4b84-bc55-2817cf4b2fa4-kube-api-access-9v7s6\" (UniqueName: \"kubernetes.io/projected/8115d771-05bd-4b84-bc55-2817cf4b2fa4-kube-api-access-9v7s6\") pod \"8115d771-05bd-4b84-bc55-2817cf4b2fa4\" (UID: \"8115d771-05bd-4b84-bc55-2817cf4b2fa4\") " Nov 03 15:44:44.954085 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:44.954055 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8115d771-05bd-4b84-bc55-2817cf4b2fa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8115d771-05bd-4b84-bc55-2817cf4b2fa4-maas-api-xks-ca\") pod \"8115d771-05bd-4b84-bc55-2817cf4b2fa4\" (UID: \"8115d771-05bd-4b84-bc55-2817cf4b2fa4\") " Nov 03 15:44:44.954234 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:44.954172 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8115d771-05bd-4b84-bc55-2817cf4b2fa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8115d771-05bd-4b84-bc55-2817cf4b2fa4-maas-api-openshift-ca\") pod \"8115d771-05bd-4b84-bc55-2817cf4b2fa4\" (UID: \"8115d771-05bd-4b84-bc55-2817cf4b2fa4\") " Nov 03 15:44:44.954784 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:44.954757 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8115d771-05bd-4b84-bc55-2817cf4b2fa4-maas-api-openshift-ca" pod "8115d771-05bd-4b84-bc55-2817cf4b2fa4" (UID: "8115d771-05bd-4b84-bc55-2817cf4b2fa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:44:44.955178 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:44.955156 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8115d771-05bd-4b84-bc55-2817cf4b2fa4-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:44.960193 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:44.957445 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8115d771-05bd-4b84-bc55-2817cf4b2fa4-maas-api-xks-ca" pod "8115d771-05bd-4b84-bc55-2817cf4b2fa4" (UID: "8115d771-05bd-4b84-bc55-2817cf4b2fa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:44:44.963920 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:44.963868 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8115d771-05bd-4b84-bc55-2817cf4b2fa4-kube-api-access-9v7s6" pod "8115d771-05bd-4b84-bc55-2817cf4b2fa4" (UID: "8115d771-05bd-4b84-bc55-2817cf4b2fa4"). InnerVolumeSpecName "kube-api-access-9v7s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:44:45.057198 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:45.057165 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v7s6\" (UniqueName: \"kubernetes.io/projected/8115d771-05bd-4b84-bc55-2817cf4b2fa4-kube-api-access-9v7s6\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:45.057198 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:45.057194 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8115d771-05bd-4b84-bc55-2817cf4b2fa4-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:45.654893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:45.654860 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" Nov 03 15:44:45.654893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:45.654880 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4" event={"ID":"8115d771-05bd-4b84-bc55-2817cf4b2fa4","Type":"ContainerDied","Data":"7844b83affcc22246f624951c1134457e98f2ec32dd7164a4fcba65ce4d63876"} Nov 03 15:44:45.655179 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:45.654925 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7844b83affcc22246f624951c1134457e98f2ec32dd7164a4fcba65ce4d63876" Nov 03 15:44:52.588805 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:52.588769 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m"] Nov 03 15:44:52.637433 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:52.637398 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d3cde-fgp4m"] Nov 03 15:44:52.681789 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:44:52.681760 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb83d831-2143-4f54-aa05-6fad48074acf" path="/var/lib/kubelet/pods/cb83d831-2143-4f54-aa05-6fad48074acf/volumes" Nov 03 15:45:00.180394 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.180264 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7"] Nov 03 15:45:00.181499 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.181472 4290 cpu_manager.go:414] "containerMap: removing container" podUID="8115d771-05bd-4b84-bc55-2817cf4b2fa4" containerName="revoke-keys" Nov 03 15:45:00.181688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.181654 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8115d771-05bd-4b84-bc55-2817cf4b2fa4" containerName="revoke-keys" Nov 03 15:45:00.182354 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.182330 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="8115d771-05bd-4b84-bc55-2817cf4b2fa4" containerName="revoke-keys" Nov 03 15:45:00.187612 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.187589 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" Nov 03 15:45:00.187946 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.187919 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm"] Nov 03 15:45:00.193278 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.193258 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" Nov 03 15:45:00.221932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.221902 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7"] Nov 03 15:45:00.227758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.227735 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" Nov 03 15:45:00.228372 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.228342 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g"] Nov 03 15:45:00.232885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.232864 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" Nov 03 15:45:00.233411 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.233392 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx"] Nov 03 15:45:00.239055 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.239033 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" Nov 03 15:45:00.244838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.244813 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw"] Nov 03 15:45:00.250505 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.250479 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7"] Nov 03 15:45:00.250670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.250573 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" Nov 03 15:45:00.289217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.289174 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:45:00.290796 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.290657 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2094a7-5e86-43c9-8e0c-917426278359-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm\" (UID: \"7d2094a7-5e86-43c9-8e0c-917426278359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" Nov 03 15:45:00.291590 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.290994 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpkp4\" (UniqueName: \"kubernetes.io/projected/7d2094a7-5e86-43c9-8e0c-917426278359-kube-api-access-jpkp4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm\" (UID: \"7d2094a7-5e86-43c9-8e0c-917426278359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" Nov 03 15:45:00.291590 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.291169 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94mdx\" (UniqueName: \"kubernetes.io/projected/be07c854-6c6c-4760-a22e-7b28af22ae21-kube-api-access-94mdx\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7\" (UID: \"be07c854-6c6c-4760-a22e-7b28af22ae21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" Nov 03 15:45:00.291990 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.291788 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be07c854-6c6c-4760-a22e-7b28af22ae21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7\" (UID: \"be07c854-6c6c-4760-a22e-7b28af22ae21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" Nov 03 15:45:00.312379 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.312332 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm"] Nov 03 15:45:00.319522 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.319489 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7"] Nov 03 15:45:00.325852 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.325820 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx"] Nov 03 15:45:00.333880 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.333854 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw"] Nov 03 15:45:00.346664 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.346635 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g"] Nov 03 15:45:00.393459 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.393421 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2094a7-5e86-43c9-8e0c-917426278359-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm\" (UID: \"7d2094a7-5e86-43c9-8e0c-917426278359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" Nov 03 15:45:00.393627 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.393465 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17557e9e-462a-4464-a66b-10b0f892bec5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7\" (UID: \"17557e9e-462a-4464-a66b-10b0f892bec5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" Nov 03 15:45:00.393627 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.393485 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8548a302-2c94-4f27-89c0-653e460d571a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g\" (UID: \"8548a302-2c94-4f27-89c0-653e460d571a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" Nov 03 15:45:00.393627 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.393583 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/321f6b11-c456-4a61-9649-b3a37398086f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw\" (UID: \"321f6b11-c456-4a61-9649-b3a37398086f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" Nov 03 15:45:00.393838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.393628 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkp4\" (UniqueName: \"kubernetes.io/projected/7d2094a7-5e86-43c9-8e0c-917426278359-kube-api-access-jpkp4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm\" (UID: \"7d2094a7-5e86-43c9-8e0c-917426278359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" Nov 03 15:45:00.393838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.393730 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhfrs\" (UniqueName: \"kubernetes.io/projected/17557e9e-462a-4464-a66b-10b0f892bec5-kube-api-access-bhfrs\") pod \"maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7\" (UID: \"17557e9e-462a-4464-a66b-10b0f892bec5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" Nov 03 15:45:00.393838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.393795 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94mdx\" (UniqueName: \"kubernetes.io/projected/be07c854-6c6c-4760-a22e-7b28af22ae21-kube-api-access-94mdx\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7\" (UID: \"be07c854-6c6c-4760-a22e-7b28af22ae21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" Nov 03 15:45:00.394023 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.393983 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fn8b\" (UniqueName: \"kubernetes.io/projected/8548a302-2c94-4f27-89c0-653e460d571a-kube-api-access-4fn8b\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g\" (UID: \"8548a302-2c94-4f27-89c0-653e460d571a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" Nov 03 15:45:00.394180 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.394155 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be07c854-6c6c-4760-a22e-7b28af22ae21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7\" (UID: \"be07c854-6c6c-4760-a22e-7b28af22ae21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" Nov 03 15:45:00.394280 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.394237 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e1db49-8fa1-40b7-ae7b-98c70dac0744-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx\" (UID: \"80e1db49-8fa1-40b7-ae7b-98c70dac0744\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" Nov 03 15:45:00.394361 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.394296 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sddmr\" (UniqueName: \"kubernetes.io/projected/80e1db49-8fa1-40b7-ae7b-98c70dac0744-kube-api-access-sddmr\") pod \"maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx\" (UID: \"80e1db49-8fa1-40b7-ae7b-98c70dac0744\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" Nov 03 15:45:00.394361 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.394323 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czbkz\" (UniqueName: \"kubernetes.io/projected/321f6b11-c456-4a61-9649-b3a37398086f-kube-api-access-czbkz\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw\" (UID: \"321f6b11-c456-4a61-9649-b3a37398086f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" Nov 03 15:45:00.394361 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.394323 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2094a7-5e86-43c9-8e0c-917426278359-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm\" (UID: \"7d2094a7-5e86-43c9-8e0c-917426278359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" Nov 03 15:45:00.394988 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.394962 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be07c854-6c6c-4760-a22e-7b28af22ae21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7\" (UID: \"be07c854-6c6c-4760-a22e-7b28af22ae21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" Nov 03 15:45:00.458778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.458696 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpkp4\" (UniqueName: \"kubernetes.io/projected/7d2094a7-5e86-43c9-8e0c-917426278359-kube-api-access-jpkp4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm\" (UID: \"7d2094a7-5e86-43c9-8e0c-917426278359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" Nov 03 15:45:00.458778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.458715 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94mdx\" (UniqueName: \"kubernetes.io/projected/be07c854-6c6c-4760-a22e-7b28af22ae21-kube-api-access-94mdx\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7\" (UID: \"be07c854-6c6c-4760-a22e-7b28af22ae21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" Nov 03 15:45:00.496153 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.496116 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sddmr\" (UniqueName: \"kubernetes.io/projected/80e1db49-8fa1-40b7-ae7b-98c70dac0744-kube-api-access-sddmr\") pod \"maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx\" (UID: \"80e1db49-8fa1-40b7-ae7b-98c70dac0744\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" Nov 03 15:45:00.496153 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.496153 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czbkz\" (UniqueName: \"kubernetes.io/projected/321f6b11-c456-4a61-9649-b3a37398086f-kube-api-access-czbkz\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw\" (UID: \"321f6b11-c456-4a61-9649-b3a37398086f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" Nov 03 15:45:00.496360 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.496227 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17557e9e-462a-4464-a66b-10b0f892bec5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7\" (UID: \"17557e9e-462a-4464-a66b-10b0f892bec5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" Nov 03 15:45:00.496360 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.496252 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8548a302-2c94-4f27-89c0-653e460d571a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g\" (UID: \"8548a302-2c94-4f27-89c0-653e460d571a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" Nov 03 15:45:00.496360 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.496284 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/321f6b11-c456-4a61-9649-b3a37398086f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw\" (UID: \"321f6b11-c456-4a61-9649-b3a37398086f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" Nov 03 15:45:00.496509 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.496366 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhfrs\" (UniqueName: \"kubernetes.io/projected/17557e9e-462a-4464-a66b-10b0f892bec5-kube-api-access-bhfrs\") pod \"maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7\" (UID: \"17557e9e-462a-4464-a66b-10b0f892bec5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" Nov 03 15:45:00.496562 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.496516 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fn8b\" (UniqueName: \"kubernetes.io/projected/8548a302-2c94-4f27-89c0-653e460d571a-kube-api-access-4fn8b\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g\" (UID: \"8548a302-2c94-4f27-89c0-653e460d571a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" Nov 03 15:45:00.496649 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.496634 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e1db49-8fa1-40b7-ae7b-98c70dac0744-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx\" (UID: \"80e1db49-8fa1-40b7-ae7b-98c70dac0744\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" Nov 03 15:45:00.496882 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.496861 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17557e9e-462a-4464-a66b-10b0f892bec5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7\" (UID: \"17557e9e-462a-4464-a66b-10b0f892bec5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" Nov 03 15:45:00.497021 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.496999 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8548a302-2c94-4f27-89c0-653e460d571a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g\" (UID: \"8548a302-2c94-4f27-89c0-653e460d571a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" Nov 03 15:45:00.497077 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.497061 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/321f6b11-c456-4a61-9649-b3a37398086f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw\" (UID: \"321f6b11-c456-4a61-9649-b3a37398086f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" Nov 03 15:45:00.497196 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.497179 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e1db49-8fa1-40b7-ae7b-98c70dac0744-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx\" (UID: \"80e1db49-8fa1-40b7-ae7b-98c70dac0744\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" Nov 03 15:45:00.504770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.504754 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" Nov 03 15:45:00.512059 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.512039 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" Nov 03 15:45:00.573582 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.573518 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czbkz\" (UniqueName: \"kubernetes.io/projected/321f6b11-c456-4a61-9649-b3a37398086f-kube-api-access-czbkz\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw\" (UID: \"321f6b11-c456-4a61-9649-b3a37398086f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" Nov 03 15:45:00.577824 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.576785 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sddmr\" (UniqueName: \"kubernetes.io/projected/80e1db49-8fa1-40b7-ae7b-98c70dac0744-kube-api-access-sddmr\") pod \"maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx\" (UID: \"80e1db49-8fa1-40b7-ae7b-98c70dac0744\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" Nov 03 15:45:00.578568 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.578516 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fn8b\" (UniqueName: \"kubernetes.io/projected/8548a302-2c94-4f27-89c0-653e460d571a-kube-api-access-4fn8b\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g\" (UID: \"8548a302-2c94-4f27-89c0-653e460d571a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" Nov 03 15:45:00.618510 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.617509 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" Nov 03 15:45:00.627607 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.627410 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" Nov 03 15:45:00.628013 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.627955 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhfrs\" (UniqueName: \"kubernetes.io/projected/17557e9e-462a-4464-a66b-10b0f892bec5-kube-api-access-bhfrs\") pod \"maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7\" (UID: \"17557e9e-462a-4464-a66b-10b0f892bec5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" Nov 03 15:45:00.640981 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.640955 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" Nov 03 15:45:00.848085 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.847993 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" Nov 03 15:45:00.952938 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:45:00.952861 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2094a7_5e86_43c9_8e0c_917426278359.slice/crio-a4e7f2f94401cc5d14b48b6bd41a4603fef8088ae2f370a2e195b7af0d0338f8 WatchSource:0}: Error finding container a4e7f2f94401cc5d14b48b6bd41a4603fef8088ae2f370a2e195b7af0d0338f8: Status 404 returned error can't find the container with id a4e7f2f94401cc5d14b48b6bd41a4603fef8088ae2f370a2e195b7af0d0338f8 Nov 03 15:45:00.985581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:00.985547 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm"] Nov 03 15:45:01.002147 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:45:01.002101 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe07c854_6c6c_4760_a22e_7b28af22ae21.slice/crio-501e033a6c306669e377b93152f34470653bd4a043dc19547ddcd49c85920510 WatchSource:0}: Error finding container 501e033a6c306669e377b93152f34470653bd4a043dc19547ddcd49c85920510: Status 404 returned error can't find the container with id 501e033a6c306669e377b93152f34470653bd4a043dc19547ddcd49c85920510 Nov 03 15:45:01.022028 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.021258 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7"] Nov 03 15:45:01.240383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.240356 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g"] Nov 03 15:45:01.244423 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:45:01.244383 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8548a302_2c94_4f27_89c0_653e460d571a.slice/crio-10ba5411e00fba814feeffb8735ce83548f738c4922a1a509df87d5ceaab0de1 WatchSource:0}: Error finding container 10ba5411e00fba814feeffb8735ce83548f738c4922a1a509df87d5ceaab0de1: Status 404 returned error can't find the container with id 10ba5411e00fba814feeffb8735ce83548f738c4922a1a509df87d5ceaab0de1 Nov 03 15:45:01.247968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.247937 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx"] Nov 03 15:45:01.288039 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.288014 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw"] Nov 03 15:45:01.291460 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:45:01.291432 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321f6b11_c456_4a61_9649_b3a37398086f.slice/crio-c7bb3769f2d7a0475f056895f53c203697e889e061df04e69603463433b2252c WatchSource:0}: Error finding container c7bb3769f2d7a0475f056895f53c203697e889e061df04e69603463433b2252c: Status 404 returned error can't find the container with id c7bb3769f2d7a0475f056895f53c203697e889e061df04e69603463433b2252c Nov 03 15:45:01.427487 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.427461 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7"] Nov 03 15:45:01.859364 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.858966 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" event={"ID":"321f6b11-c456-4a61-9649-b3a37398086f","Type":"ContainerStarted","Data":"a8775bd6a5d14f70f2f9f88606c3ef44c9c0165c845373ad6da10545183d406b"} Nov 03 15:45:01.859364 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.859030 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" event={"ID":"321f6b11-c456-4a61-9649-b3a37398086f","Type":"ContainerStarted","Data":"c7bb3769f2d7a0475f056895f53c203697e889e061df04e69603463433b2252c"} Nov 03 15:45:01.861094 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.861060 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" event={"ID":"80e1db49-8fa1-40b7-ae7b-98c70dac0744","Type":"ContainerStarted","Data":"bdae0bac039dba15fd13b160cc9258c1e1ec9e1e35d19dbb1219b5dd9c783ef8"} Nov 03 15:45:01.861223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.861121 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" event={"ID":"80e1db49-8fa1-40b7-ae7b-98c70dac0744","Type":"ContainerStarted","Data":"63ea536d52dc33ba196ea17cc22e2bcf0f871f3b73daec6a12f863fbc64a0e6c"} Nov 03 15:45:01.863179 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.863144 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" event={"ID":"be07c854-6c6c-4760-a22e-7b28af22ae21","Type":"ContainerStarted","Data":"3f1758a5231d042d77d31371bbefe9fdadddd6e55900d16751a1e3b4806b6760"} Nov 03 15:45:01.863297 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.863204 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" event={"ID":"be07c854-6c6c-4760-a22e-7b28af22ae21","Type":"ContainerStarted","Data":"501e033a6c306669e377b93152f34470653bd4a043dc19547ddcd49c85920510"} Nov 03 15:45:01.864970 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.864947 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" event={"ID":"7d2094a7-5e86-43c9-8e0c-917426278359","Type":"ContainerStarted","Data":"299b1ca66be3651b128abdcda69d0bf2b31988ba26c80567187776697351586c"} Nov 03 15:45:01.865077 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.864994 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" event={"ID":"7d2094a7-5e86-43c9-8e0c-917426278359","Type":"ContainerStarted","Data":"a4e7f2f94401cc5d14b48b6bd41a4603fef8088ae2f370a2e195b7af0d0338f8"} Nov 03 15:45:01.866430 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.866407 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" event={"ID":"17557e9e-462a-4464-a66b-10b0f892bec5","Type":"ContainerStarted","Data":"186e93ecbedd406154f39d7387ad08f31c9d8a394a63d3166bc3094248988f0f"} Nov 03 15:45:01.868302 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.868243 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" event={"ID":"8548a302-2c94-4f27-89c0-653e460d571a","Type":"ContainerStarted","Data":"37cad4387016f1c793eb68f940df2fb65cceb1b6825d295283f1975ad9e10b1d"} Nov 03 15:45:01.868302 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.868278 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" event={"ID":"8548a302-2c94-4f27-89c0-653e460d571a","Type":"ContainerStarted","Data":"10ba5411e00fba814feeffb8735ce83548f738c4922a1a509df87d5ceaab0de1"} Nov 03 15:45:01.980057 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:01.980006 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" podStartSLOduration=1.979992277 podStartE2EDuration="1.979992277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:01.975831674 +0000 UTC m=+2209.406789497" watchObservedRunningTime="2026-11-03 15:45:01.979992277 +0000 UTC m=+2209.410950082" Nov 03 15:45:02.133230 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.133166 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" podStartSLOduration=2.133151299 podStartE2EDuration="2.133151299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:02.130232368 +0000 UTC m=+2209.561190178" watchObservedRunningTime="2026-11-03 15:45:02.133151299 +0000 UTC m=+2209.564109118" Nov 03 15:45:02.266068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.266010 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" podStartSLOduration=2.265995545 podStartE2EDuration="2.265995545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:02.261066368 +0000 UTC m=+2209.692024172" watchObservedRunningTime="2026-11-03 15:45:02.265995545 +0000 UTC m=+2209.696953350" Nov 03 15:45:02.435495 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.435446 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" podStartSLOduration=2.435422111 podStartE2EDuration="2.435422111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:02.431101702 +0000 UTC m=+2209.862059517" watchObservedRunningTime="2026-11-03 15:45:02.435422111 +0000 UTC m=+2209.866379916" Nov 03 15:45:02.550767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.550671 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" podStartSLOduration=2.550652935 podStartE2EDuration="2.550652935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:02.549592324 +0000 UTC m=+2209.980550130" watchObservedRunningTime="2026-11-03 15:45:02.550652935 +0000 UTC m=+2209.981610742" Nov 03 15:45:02.878912 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.878878 4290 generic.go:356] "Generic (PLEG): container finished" podID="321f6b11-c456-4a61-9649-b3a37398086f" containerID="a8775bd6a5d14f70f2f9f88606c3ef44c9c0165c845373ad6da10545183d406b" exitCode=60 Nov 03 15:45:02.879105 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.878957 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" event={"ID":"321f6b11-c456-4a61-9649-b3a37398086f","Type":"ContainerDied","Data":"a8775bd6a5d14f70f2f9f88606c3ef44c9c0165c845373ad6da10545183d406b"} Nov 03 15:45:02.879276 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.879252 4290 scope.go:122] "RemoveContainer" containerID="a8775bd6a5d14f70f2f9f88606c3ef44c9c0165c845373ad6da10545183d406b" Nov 03 15:45:02.880934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.880909 4290 generic.go:356] "Generic (PLEG): container finished" podID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" containerID="bdae0bac039dba15fd13b160cc9258c1e1ec9e1e35d19dbb1219b5dd9c783ef8" exitCode=60 Nov 03 15:45:02.881051 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.881006 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" event={"ID":"80e1db49-8fa1-40b7-ae7b-98c70dac0744","Type":"ContainerDied","Data":"bdae0bac039dba15fd13b160cc9258c1e1ec9e1e35d19dbb1219b5dd9c783ef8"} Nov 03 15:45:02.881272 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.881252 4290 scope.go:122] "RemoveContainer" containerID="bdae0bac039dba15fd13b160cc9258c1e1ec9e1e35d19dbb1219b5dd9c783ef8" Nov 03 15:45:02.884272 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.883143 4290 generic.go:356] "Generic (PLEG): container finished" podID="be07c854-6c6c-4760-a22e-7b28af22ae21" containerID="3f1758a5231d042d77d31371bbefe9fdadddd6e55900d16751a1e3b4806b6760" exitCode=60 Nov 03 15:45:02.884272 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.883277 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" event={"ID":"be07c854-6c6c-4760-a22e-7b28af22ae21","Type":"ContainerDied","Data":"3f1758a5231d042d77d31371bbefe9fdadddd6e55900d16751a1e3b4806b6760"} Nov 03 15:45:02.884272 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.883481 4290 scope.go:122] "RemoveContainer" containerID="3f1758a5231d042d77d31371bbefe9fdadddd6e55900d16751a1e3b4806b6760" Nov 03 15:45:02.888600 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.888578 4290 generic.go:356] "Generic (PLEG): container finished" podID="7d2094a7-5e86-43c9-8e0c-917426278359" containerID="299b1ca66be3651b128abdcda69d0bf2b31988ba26c80567187776697351586c" exitCode=60 Nov 03 15:45:02.888687 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.888659 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" event={"ID":"7d2094a7-5e86-43c9-8e0c-917426278359","Type":"ContainerDied","Data":"299b1ca66be3651b128abdcda69d0bf2b31988ba26c80567187776697351586c"} Nov 03 15:45:02.888922 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.888908 4290 scope.go:122] "RemoveContainer" containerID="299b1ca66be3651b128abdcda69d0bf2b31988ba26c80567187776697351586c" Nov 03 15:45:02.890528 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.890462 4290 generic.go:356] "Generic (PLEG): container finished" podID="17557e9e-462a-4464-a66b-10b0f892bec5" containerID="faaf0db670c6e3098cd20dd627c35a1c17dc67f1785acb938b9bca3a35e01931" exitCode=60 Nov 03 15:45:02.890528 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.890516 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" event={"ID":"17557e9e-462a-4464-a66b-10b0f892bec5","Type":"ContainerDied","Data":"faaf0db670c6e3098cd20dd627c35a1c17dc67f1785acb938b9bca3a35e01931"} Nov 03 15:45:02.890858 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.890841 4290 scope.go:122] "RemoveContainer" containerID="faaf0db670c6e3098cd20dd627c35a1c17dc67f1785acb938b9bca3a35e01931" Nov 03 15:45:02.892329 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.892308 4290 generic.go:356] "Generic (PLEG): container finished" podID="8548a302-2c94-4f27-89c0-653e460d571a" containerID="37cad4387016f1c793eb68f940df2fb65cceb1b6825d295283f1975ad9e10b1d" exitCode=60 Nov 03 15:45:02.892421 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.892390 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" event={"ID":"8548a302-2c94-4f27-89c0-653e460d571a","Type":"ContainerDied","Data":"37cad4387016f1c793eb68f940df2fb65cceb1b6825d295283f1975ad9e10b1d"} Nov 03 15:45:02.892573 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:02.892558 4290 scope.go:122] "RemoveContainer" containerID="37cad4387016f1c793eb68f940df2fb65cceb1b6825d295283f1975ad9e10b1d" Nov 03 15:45:03.902067 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:03.902017 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" event={"ID":"7d2094a7-5e86-43c9-8e0c-917426278359","Type":"ContainerStarted","Data":"f6be744c89a732965fe238fe6a1fee8b93c07f71411e9d531201e47e6dfaa4c7"} Nov 03 15:45:03.904121 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:03.904097 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" event={"ID":"17557e9e-462a-4464-a66b-10b0f892bec5","Type":"ContainerStarted","Data":"6c6e95cc345245e85e6f7a4be50577e760ee7eb16126da4dd118d4b029e670ae"} Nov 03 15:45:03.906040 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:03.906018 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" event={"ID":"8548a302-2c94-4f27-89c0-653e460d571a","Type":"ContainerStarted","Data":"612956036e8dcb40dfd70e2d08644ec57295e54b531e9b59119d9c36821400c1"} Nov 03 15:45:03.907891 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:03.907871 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" event={"ID":"321f6b11-c456-4a61-9649-b3a37398086f","Type":"ContainerStarted","Data":"65c5a5278f6d0ba87b4f2b3ff4b34a8a30b82ecb377e64fd69a01bd574a10720"} Nov 03 15:45:03.909835 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:03.909814 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" event={"ID":"80e1db49-8fa1-40b7-ae7b-98c70dac0744","Type":"ContainerStarted","Data":"3e20f721b8a7fa8a7b4f8f7bf1c2f97674271690020666165476edcd05d7da9b"} Nov 03 15:45:03.911653 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:03.911635 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" event={"ID":"be07c854-6c6c-4760-a22e-7b28af22ae21","Type":"ContainerStarted","Data":"889d0d353f95ecc558de7d8c5053211b07ef69cafef1a459ce4b41425fb00114"} Nov 03 15:45:04.135341 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.135283 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" podStartSLOduration=4.135268245 podStartE2EDuration="4.135268245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:04.134473441 +0000 UTC m=+2211.565431246" watchObservedRunningTime="2026-11-03 15:45:04.135268245 +0000 UTC m=+2211.566226048" Nov 03 15:45:04.922042 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.922009 4290 generic.go:356] "Generic (PLEG): container finished" podID="be07c854-6c6c-4760-a22e-7b28af22ae21" containerID="889d0d353f95ecc558de7d8c5053211b07ef69cafef1a459ce4b41425fb00114" exitCode=60 Nov 03 15:45:04.922497 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.922093 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" event={"ID":"be07c854-6c6c-4760-a22e-7b28af22ae21","Type":"ContainerDied","Data":"889d0d353f95ecc558de7d8c5053211b07ef69cafef1a459ce4b41425fb00114"} Nov 03 15:45:04.922497 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.922156 4290 scope.go:122] "RemoveContainer" containerID="3f1758a5231d042d77d31371bbefe9fdadddd6e55900d16751a1e3b4806b6760" Nov 03 15:45:04.922497 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.922356 4290 scope.go:122] "RemoveContainer" containerID="889d0d353f95ecc558de7d8c5053211b07ef69cafef1a459ce4b41425fb00114" Nov 03 15:45:04.922790 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:04.922763 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7_odh-ai-gateway-infra(be07c854-6c6c-4760-a22e-7b28af22ae21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" podUID="be07c854-6c6c-4760-a22e-7b28af22ae21" Nov 03 15:45:04.924382 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.924361 4290 generic.go:356] "Generic (PLEG): container finished" podID="7d2094a7-5e86-43c9-8e0c-917426278359" containerID="f6be744c89a732965fe238fe6a1fee8b93c07f71411e9d531201e47e6dfaa4c7" exitCode=60 Nov 03 15:45:04.924488 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.924435 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" event={"ID":"7d2094a7-5e86-43c9-8e0c-917426278359","Type":"ContainerDied","Data":"f6be744c89a732965fe238fe6a1fee8b93c07f71411e9d531201e47e6dfaa4c7"} Nov 03 15:45:04.924664 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.924647 4290 scope.go:122] "RemoveContainer" containerID="f6be744c89a732965fe238fe6a1fee8b93c07f71411e9d531201e47e6dfaa4c7" Nov 03 15:45:04.924988 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:04.924964 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm_odh-ai-gateway-infra(7d2094a7-5e86-43c9-8e0c-917426278359)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" podUID="7d2094a7-5e86-43c9-8e0c-917426278359" Nov 03 15:45:04.926636 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.926617 4290 generic.go:356] "Generic (PLEG): container finished" podID="17557e9e-462a-4464-a66b-10b0f892bec5" containerID="6c6e95cc345245e85e6f7a4be50577e760ee7eb16126da4dd118d4b029e670ae" exitCode=60 Nov 03 15:45:04.926738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.926666 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" event={"ID":"17557e9e-462a-4464-a66b-10b0f892bec5","Type":"ContainerDied","Data":"6c6e95cc345245e85e6f7a4be50577e760ee7eb16126da4dd118d4b029e670ae"} Nov 03 15:45:04.926963 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.926949 4290 scope.go:122] "RemoveContainer" containerID="6c6e95cc345245e85e6f7a4be50577e760ee7eb16126da4dd118d4b029e670ae" Nov 03 15:45:04.927243 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:04.927216 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7_odh-ai-gateway-infra(17557e9e-462a-4464-a66b-10b0f892bec5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" podUID="17557e9e-462a-4464-a66b-10b0f892bec5" Nov 03 15:45:04.928769 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.928745 4290 generic.go:356] "Generic (PLEG): container finished" podID="8548a302-2c94-4f27-89c0-653e460d571a" containerID="612956036e8dcb40dfd70e2d08644ec57295e54b531e9b59119d9c36821400c1" exitCode=60 Nov 03 15:45:04.928859 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.928817 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" event={"ID":"8548a302-2c94-4f27-89c0-653e460d571a","Type":"ContainerDied","Data":"612956036e8dcb40dfd70e2d08644ec57295e54b531e9b59119d9c36821400c1"} Nov 03 15:45:04.929052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.929028 4290 scope.go:122] "RemoveContainer" containerID="612956036e8dcb40dfd70e2d08644ec57295e54b531e9b59119d9c36821400c1" Nov 03 15:45:04.929299 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:04.929281 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g_odh-ai-gateway-infra(8548a302-2c94-4f27-89c0-653e460d571a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" podUID="8548a302-2c94-4f27-89c0-653e460d571a" Nov 03 15:45:04.930972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.930950 4290 generic.go:356] "Generic (PLEG): container finished" podID="321f6b11-c456-4a61-9649-b3a37398086f" containerID="65c5a5278f6d0ba87b4f2b3ff4b34a8a30b82ecb377e64fd69a01bd574a10720" exitCode=60 Nov 03 15:45:04.931066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.930978 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" event={"ID":"321f6b11-c456-4a61-9649-b3a37398086f","Type":"ContainerDied","Data":"65c5a5278f6d0ba87b4f2b3ff4b34a8a30b82ecb377e64fd69a01bd574a10720"} Nov 03 15:45:04.931241 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.931223 4290 scope.go:122] "RemoveContainer" containerID="65c5a5278f6d0ba87b4f2b3ff4b34a8a30b82ecb377e64fd69a01bd574a10720" Nov 03 15:45:04.931596 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:04.931575 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw_odh-ai-gateway-infra(321f6b11-c456-4a61-9649-b3a37398086f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" podUID="321f6b11-c456-4a61-9649-b3a37398086f" Nov 03 15:45:04.932995 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.932978 4290 generic.go:356] "Generic (PLEG): container finished" podID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" containerID="3e20f721b8a7fa8a7b4f8f7bf1c2f97674271690020666165476edcd05d7da9b" exitCode=60 Nov 03 15:45:04.933089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.933008 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" event={"ID":"80e1db49-8fa1-40b7-ae7b-98c70dac0744","Type":"ContainerDied","Data":"3e20f721b8a7fa8a7b4f8f7bf1c2f97674271690020666165476edcd05d7da9b"} Nov 03 15:45:04.933213 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.933199 4290 scope.go:122] "RemoveContainer" containerID="3e20f721b8a7fa8a7b4f8f7bf1c2f97674271690020666165476edcd05d7da9b" Nov 03 15:45:04.933422 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:04.933406 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx_odh-ai-gateway-infra(80e1db49-8fa1-40b7-ae7b-98c70dac0744)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" podUID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" Nov 03 15:45:04.942613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.942587 4290 scope.go:122] "RemoveContainer" containerID="299b1ca66be3651b128abdcda69d0bf2b31988ba26c80567187776697351586c" Nov 03 15:45:04.957896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.957870 4290 scope.go:122] "RemoveContainer" containerID="faaf0db670c6e3098cd20dd627c35a1c17dc67f1785acb938b9bca3a35e01931" Nov 03 15:45:04.974502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.974481 4290 scope.go:122] "RemoveContainer" containerID="37cad4387016f1c793eb68f940df2fb65cceb1b6825d295283f1975ad9e10b1d" Nov 03 15:45:04.990246 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:04.990224 4290 scope.go:122] "RemoveContainer" containerID="a8775bd6a5d14f70f2f9f88606c3ef44c9c0165c845373ad6da10545183d406b" Nov 03 15:45:05.004077 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:05.004057 4290 scope.go:122] "RemoveContainer" containerID="bdae0bac039dba15fd13b160cc9258c1e1ec9e1e35d19dbb1219b5dd9c783ef8" Nov 03 15:45:05.945170 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:05.945139 4290 scope.go:122] "RemoveContainer" containerID="65c5a5278f6d0ba87b4f2b3ff4b34a8a30b82ecb377e64fd69a01bd574a10720" Nov 03 15:45:05.945668 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:05.945553 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw_odh-ai-gateway-infra(321f6b11-c456-4a61-9649-b3a37398086f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" podUID="321f6b11-c456-4a61-9649-b3a37398086f" Nov 03 15:45:05.947081 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:05.947059 4290 scope.go:122] "RemoveContainer" containerID="3e20f721b8a7fa8a7b4f8f7bf1c2f97674271690020666165476edcd05d7da9b" Nov 03 15:45:05.947380 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:05.947360 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx_odh-ai-gateway-infra(80e1db49-8fa1-40b7-ae7b-98c70dac0744)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" podUID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" Nov 03 15:45:05.949076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:05.949041 4290 scope.go:122] "RemoveContainer" containerID="889d0d353f95ecc558de7d8c5053211b07ef69cafef1a459ce4b41425fb00114" Nov 03 15:45:05.949384 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:05.949361 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7_odh-ai-gateway-infra(be07c854-6c6c-4760-a22e-7b28af22ae21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" podUID="be07c854-6c6c-4760-a22e-7b28af22ae21" Nov 03 15:45:05.950931 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:05.950910 4290 scope.go:122] "RemoveContainer" containerID="f6be744c89a732965fe238fe6a1fee8b93c07f71411e9d531201e47e6dfaa4c7" Nov 03 15:45:05.951235 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:05.951212 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm_odh-ai-gateway-infra(7d2094a7-5e86-43c9-8e0c-917426278359)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" podUID="7d2094a7-5e86-43c9-8e0c-917426278359" Nov 03 15:45:05.952690 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:05.952668 4290 scope.go:122] "RemoveContainer" containerID="6c6e95cc345245e85e6f7a4be50577e760ee7eb16126da4dd118d4b029e670ae" Nov 03 15:45:05.953016 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:05.952980 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7_odh-ai-gateway-infra(17557e9e-462a-4464-a66b-10b0f892bec5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" podUID="17557e9e-462a-4464-a66b-10b0f892bec5" Nov 03 15:45:05.954439 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:05.954422 4290 scope.go:122] "RemoveContainer" containerID="612956036e8dcb40dfd70e2d08644ec57295e54b531e9b59119d9c36821400c1" Nov 03 15:45:05.954742 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:05.954686 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g_odh-ai-gateway-infra(8548a302-2c94-4f27-89c0-653e460d571a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" podUID="8548a302-2c94-4f27-89c0-653e460d571a" Nov 03 15:45:06.043113 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.042033 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4"] Nov 03 15:45:06.049288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.049266 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.109231 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.109202 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4"] Nov 03 15:45:06.232935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.232852 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.233071 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.232985 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.233071 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.233010 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.233071 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.233034 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.233194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.233135 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.233194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.233169 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94vzv\" (UniqueName: \"kubernetes.io/projected/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-kube-api-access-94vzv\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.233257 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.233196 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.233331 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.233320 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.233378 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.233366 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.335203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.335169 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.335374 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.335266 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.335374 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.335313 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.335527 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.335427 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.335581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.335536 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.335581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.335563 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.335672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.335585 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.335672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.335654 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.335805 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.335678 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.335805 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.335682 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94vzv\" (UniqueName: \"kubernetes.io/projected/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-kube-api-access-94vzv\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.335904 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.335798 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.336055 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.336029 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.336186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.336113 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.336186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.336174 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.338189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.338165 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.338189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.338178 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.397402 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.397368 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.397536 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.397504 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94vzv\" (UniqueName: \"kubernetes.io/projected/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-kube-api-access-94vzv\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:06.673929 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:06.673896 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:07.124672 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:45:07.124641 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb1b2ea_09d4_4ce3_ab97_5d1c48c4c27c.slice/crio-6fdb33126da2cf5abea89a223f813e7463650e0c90195017e4ee0cb136b95060 WatchSource:0}: Error finding container 6fdb33126da2cf5abea89a223f813e7463650e0c90195017e4ee0cb136b95060: Status 404 returned error can't find the container with id 6fdb33126da2cf5abea89a223f813e7463650e0c90195017e4ee0cb136b95060 Nov 03 15:45:07.126231 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:07.126205 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4"] Nov 03 15:45:07.127809 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:07.127778 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:45:07.127890 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:07.127838 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:45:07.127890 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:07.127867 4290 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 15:45:08.009565 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:08.009528 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" event={"ID":"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c","Type":"ContainerStarted","Data":"546aa4269313490c85c3b33ffa9e7d1e1f3fccaa77832c4a84221895d641ebbd"} Nov 03 15:45:08.009812 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:08.009582 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" event={"ID":"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c","Type":"ContainerStarted","Data":"6fdb33126da2cf5abea89a223f813e7463650e0c90195017e4ee0cb136b95060"} Nov 03 15:45:08.186863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:08.186777 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" podStartSLOduration=3.186757181 podStartE2EDuration="3.186757181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:08.180316089 +0000 UTC m=+2215.611273899" watchObservedRunningTime="2026-11-03 15:45:08.186757181 +0000 UTC m=+2215.617714987" Nov 03 15:45:08.677218 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:08.677187 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 15:45:08.677412 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:08.677249 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" podUID="fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 15:45:08.683050 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:08.682984 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:08.722020 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:08.721985 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:45:09.674398 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:09.674368 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 15:45:09.674813 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:09.674419 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" podUID="fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 15:45:10.674679 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:10.674631 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 15:45:10.675154 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:10.674735 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" podUID="fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 15:45:11.675429 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:11.675390 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 15:45:11.675848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:11.675449 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" podUID="fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 15:45:12.230087 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:12.230052 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf"] Nov 03 15:45:12.372060 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:12.372021 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h"] Nov 03 15:45:12.727727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:12.727285 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 15:45:12.727727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:12.727354 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" podUID="fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 15:45:13.654204 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:13.654166 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7310-1a3dbd"] Nov 03 15:45:13.660108 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:13.660085 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7310-1a3dbd" Nov 03 15:45:13.674686 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:13.674656 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 15:45:13.674842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:13.674732 4290 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" podUID="fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 15:45:13.791739 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:13.760903 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hb74t\"" Nov 03 15:45:13.832219 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:13.832182 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7310-1a3dbd"] Nov 03 15:45:13.868942 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:13.868908 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v4np\" (UniqueName: \"kubernetes.io/projected/828130bd-8ec4-42c3-8c11-c5ecd39b2a72-kube-api-access-8v4np\") pod \"test-curl-7310-1a3dbd\" (UID: \"828130bd-8ec4-42c3-8c11-c5ecd39b2a72\") " pod="opendatahub/test-curl-7310-1a3dbd" Nov 03 15:45:13.971937 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:13.970626 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v4np\" (UniqueName: \"kubernetes.io/projected/828130bd-8ec4-42c3-8c11-c5ecd39b2a72-kube-api-access-8v4np\") pod \"test-curl-7310-1a3dbd\" (UID: \"828130bd-8ec4-42c3-8c11-c5ecd39b2a72\") " pod="opendatahub/test-curl-7310-1a3dbd" Nov 03 15:45:14.032644 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:14.032603 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v4np\" (UniqueName: \"kubernetes.io/projected/828130bd-8ec4-42c3-8c11-c5ecd39b2a72-kube-api-access-8v4np\") pod \"test-curl-7310-1a3dbd\" (UID: \"828130bd-8ec4-42c3-8c11-c5ecd39b2a72\") " pod="opendatahub/test-curl-7310-1a3dbd" Nov 03 15:45:14.273875 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:14.273772 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7310-1a3dbd" Nov 03 15:45:14.671798 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:14.671772 4290 scope.go:122] "RemoveContainer" containerID="3e20f721b8a7fa8a7b4f8f7bf1c2f97674271690020666165476edcd05d7da9b" Nov 03 15:45:14.682044 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:45:14.680799 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod828130bd_8ec4_42c3_8c11_c5ecd39b2a72.slice/crio-143aaaa41e2b5fdd082523da4c46655dc4a326e2828ed2eb02c49b7986345403 WatchSource:0}: Error finding container 143aaaa41e2b5fdd082523da4c46655dc4a326e2828ed2eb02c49b7986345403: Status 404 returned error can't find the container with id 143aaaa41e2b5fdd082523da4c46655dc4a326e2828ed2eb02c49b7986345403 Nov 03 15:45:14.687049 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:14.687026 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:14.687159 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:14.687065 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7310-1a3dbd"] Nov 03 15:45:14.687539 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:14.687512 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:14.688615 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:14.688591 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:45:15.127348 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:15.127315 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7310-1a3dbd" event={"ID":"828130bd-8ec4-42c3-8c11-c5ecd39b2a72","Type":"ContainerStarted","Data":"50967f62ca308591b09442d6c5c6b38b7583ffc8d7146a457d9574195728c0d5"} Nov 03 15:45:15.127348 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:15.127352 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7310-1a3dbd" event={"ID":"828130bd-8ec4-42c3-8c11-c5ecd39b2a72","Type":"ContainerStarted","Data":"143aaaa41e2b5fdd082523da4c46655dc4a326e2828ed2eb02c49b7986345403"} Nov 03 15:45:15.246939 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:15.246819 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7310-1a3dbd" podStartSLOduration=2.246806505 podStartE2EDuration="2.246806505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:15.24112023 +0000 UTC m=+2222.672078038" watchObservedRunningTime="2026-11-03 15:45:15.246806505 +0000 UTC m=+2222.677764310" Nov 03 15:45:15.670761 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:15.670737 4290 scope.go:122] "RemoveContainer" containerID="889d0d353f95ecc558de7d8c5053211b07ef69cafef1a459ce4b41425fb00114" Nov 03 15:45:15.671102 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:15.671081 4290 scope.go:122] "RemoveContainer" containerID="f6be744c89a732965fe238fe6a1fee8b93c07f71411e9d531201e47e6dfaa4c7" Nov 03 15:45:16.138151 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:16.138122 4290 generic.go:356] "Generic (PLEG): container finished" podID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" containerID="1ee1c80d038c1ab077f4a0d99229a7e56c985addbbb7c57859a6e71ed1cb67b4" exitCode=60 Nov 03 15:45:16.138597 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:16.138202 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" event={"ID":"80e1db49-8fa1-40b7-ae7b-98c70dac0744","Type":"ContainerDied","Data":"1ee1c80d038c1ab077f4a0d99229a7e56c985addbbb7c57859a6e71ed1cb67b4"} Nov 03 15:45:16.138597 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:16.138267 4290 scope.go:122] "RemoveContainer" containerID="3e20f721b8a7fa8a7b4f8f7bf1c2f97674271690020666165476edcd05d7da9b" Nov 03 15:45:16.138787 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:16.138689 4290 scope.go:122] "RemoveContainer" containerID="1ee1c80d038c1ab077f4a0d99229a7e56c985addbbb7c57859a6e71ed1cb67b4" Nov 03 15:45:16.140882 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:16.139159 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx_odh-ai-gateway-infra(80e1db49-8fa1-40b7-ae7b-98c70dac0744)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" podUID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" Nov 03 15:45:16.670860 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:16.670833 4290 scope.go:122] "RemoveContainer" containerID="612956036e8dcb40dfd70e2d08644ec57295e54b531e9b59119d9c36821400c1" Nov 03 15:45:16.671029 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:16.670959 4290 scope.go:122] "RemoveContainer" containerID="6c6e95cc345245e85e6f7a4be50577e760ee7eb16126da4dd118d4b029e670ae" Nov 03 15:45:16.962326 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:16.962297 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7310-1a3dbd"] Nov 03 15:45:16.972679 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:16.972647 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7310-1a3dbd"] Nov 03 15:45:17.151995 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.151960 4290 generic.go:356] "Generic (PLEG): container finished" podID="be07c854-6c6c-4760-a22e-7b28af22ae21" containerID="f182e2fe4d8d6bcf76b46e98e9845b882f002777bda846f5e11d69254ddc5ced" exitCode=60 Nov 03 15:45:17.152410 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.152026 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" event={"ID":"be07c854-6c6c-4760-a22e-7b28af22ae21","Type":"ContainerDied","Data":"f182e2fe4d8d6bcf76b46e98e9845b882f002777bda846f5e11d69254ddc5ced"} Nov 03 15:45:17.152410 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.152084 4290 scope.go:122] "RemoveContainer" containerID="889d0d353f95ecc558de7d8c5053211b07ef69cafef1a459ce4b41425fb00114" Nov 03 15:45:17.152547 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.152475 4290 scope.go:122] "RemoveContainer" containerID="f182e2fe4d8d6bcf76b46e98e9845b882f002777bda846f5e11d69254ddc5ced" Nov 03 15:45:17.152907 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:17.152883 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7_odh-ai-gateway-infra(be07c854-6c6c-4760-a22e-7b28af22ae21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" podUID="be07c854-6c6c-4760-a22e-7b28af22ae21" Nov 03 15:45:17.154354 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.154334 4290 generic.go:356] "Generic (PLEG): container finished" podID="7d2094a7-5e86-43c9-8e0c-917426278359" containerID="05f4798973b46eff4e563242ac6a0a1a0cbd31e5c2587617af8f07193ed9ea1c" exitCode=60 Nov 03 15:45:17.154457 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.154382 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" event={"ID":"7d2094a7-5e86-43c9-8e0c-917426278359","Type":"ContainerDied","Data":"05f4798973b46eff4e563242ac6a0a1a0cbd31e5c2587617af8f07193ed9ea1c"} Nov 03 15:45:17.154817 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.154798 4290 scope.go:122] "RemoveContainer" containerID="05f4798973b46eff4e563242ac6a0a1a0cbd31e5c2587617af8f07193ed9ea1c" Nov 03 15:45:17.155163 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:17.155138 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm_odh-ai-gateway-infra(7d2094a7-5e86-43c9-8e0c-917426278359)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" podUID="7d2094a7-5e86-43c9-8e0c-917426278359" Nov 03 15:45:17.156414 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.156390 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" event={"ID":"17557e9e-462a-4464-a66b-10b0f892bec5","Type":"ContainerStarted","Data":"788ab550b29c372e0ceec5b32c2089c85c3b73c68407a3c92543e216027740c8"} Nov 03 15:45:17.159185 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.159162 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" event={"ID":"8548a302-2c94-4f27-89c0-653e460d571a","Type":"ContainerStarted","Data":"1c89f96ced74deed85197addf437368b043a79df66ef76d342f61269bacf2e2f"} Nov 03 15:45:17.159303 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.159287 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7310-1a3dbd" podUID="828130bd-8ec4-42c3-8c11-c5ecd39b2a72" containerName="test-curl-7310-1a3dbd" containerID="cri-o://50967f62ca308591b09442d6c5c6b38b7583ffc8d7146a457d9574195728c0d5" gracePeriod=2 Nov 03 15:45:17.171158 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.171138 4290 scope.go:122] "RemoveContainer" containerID="f6be744c89a732965fe238fe6a1fee8b93c07f71411e9d531201e47e6dfaa4c7" Nov 03 15:45:17.230755 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.230682 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" podUID="6e79fa4b-f118-4d97-a393-ad65d2c46cc6" containerName="payload-processing" containerID="cri-o://3361c095289b146d7674f7febb333596add3af550024f98d09de932ac76383e0" gracePeriod=25 Nov 03 15:45:17.301391 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.301326 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx"] Nov 03 15:45:17.329155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.329127 4290 scope.go:122] "RemoveContainer" containerID="065b9f5d2184d16541b053bc37473474da678848b312d9d67cf000a2d94577c3" Nov 03 15:45:17.373763 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.373020 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" podUID="8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705" containerName="payload-pre-processing" containerID="cri-o://67cae3c6688dca21ac410d6d4fd897a0717915c4c3623cefdadafea24f000aaf" gracePeriod=25 Nov 03 15:45:17.492983 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.492913 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" Nov 03 15:45:17.536736 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.536648 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e1db49-8fa1-40b7-ae7b-98c70dac0744-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e1db49-8fa1-40b7-ae7b-98c70dac0744-maas-api-service-ca\") pod \"80e1db49-8fa1-40b7-ae7b-98c70dac0744\" (UID: \"80e1db49-8fa1-40b7-ae7b-98c70dac0744\") " Nov 03 15:45:17.537345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.537293 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e1db49-8fa1-40b7-ae7b-98c70dac0744-maas-api-service-ca" pod "80e1db49-8fa1-40b7-ae7b-98c70dac0744" (UID: "80e1db49-8fa1-40b7-ae7b-98c70dac0744"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:17.538499 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.537481 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e1db49-8fa1-40b7-ae7b-98c70dac0744-kube-api-access-sddmr\" (UniqueName: \"kubernetes.io/projected/80e1db49-8fa1-40b7-ae7b-98c70dac0744-kube-api-access-sddmr\") pod \"80e1db49-8fa1-40b7-ae7b-98c70dac0744\" (UID: \"80e1db49-8fa1-40b7-ae7b-98c70dac0744\") " Nov 03 15:45:17.538499 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.538245 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e1db49-8fa1-40b7-ae7b-98c70dac0744-maas-api-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:17.548786 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.547851 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e1db49-8fa1-40b7-ae7b-98c70dac0744-kube-api-access-sddmr" pod "80e1db49-8fa1-40b7-ae7b-98c70dac0744" (UID: "80e1db49-8fa1-40b7-ae7b-98c70dac0744"). InnerVolumeSpecName "kube-api-access-sddmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:17.585051 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.584998 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" Nov 03 15:45:17.592636 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:17.592602 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7310-1a3dbd\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="828130bd-8ec4-42c3-8c11-c5ecd39b2a72" pod="opendatahub/test-curl-7310-1a3dbd" Nov 03 15:45:17.621255 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.621224 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj"] Nov 03 15:45:17.622058 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.622036 4290 cpu_manager.go:414] "containerMap: removing container" podUID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" containerName="cleanup" Nov 03 15:45:17.622058 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.622054 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" containerName="cleanup" Nov 03 15:45:17.622229 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.622073 4290 cpu_manager.go:414] "containerMap: removing container" podUID="6e79fa4b-f118-4d97-a393-ad65d2c46cc6" containerName="payload-processing" Nov 03 15:45:17.622229 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.622088 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e79fa4b-f118-4d97-a393-ad65d2c46cc6" containerName="payload-processing" Nov 03 15:45:17.622229 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.622116 4290 cpu_manager.go:414] "containerMap: removing container" podUID="828130bd-8ec4-42c3-8c11-c5ecd39b2a72" containerName="test-curl-7310-1a3dbd" Nov 03 15:45:17.622229 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.622121 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828130bd-8ec4-42c3-8c11-c5ecd39b2a72" containerName="test-curl-7310-1a3dbd" Nov 03 15:45:17.622229 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.622140 4290 cpu_manager.go:414] "containerMap: removing container" podUID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" containerName="cleanup" Nov 03 15:45:17.622229 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.622144 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" containerName="cleanup" Nov 03 15:45:17.622598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.622251 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" containerName="cleanup" Nov 03 15:45:17.622598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.622261 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e79fa4b-f118-4d97-a393-ad65d2c46cc6" containerName="payload-processing" Nov 03 15:45:17.622598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.622271 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="828130bd-8ec4-42c3-8c11-c5ecd39b2a72" containerName="test-curl-7310-1a3dbd" Nov 03 15:45:17.622598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.622278 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" containerName="cleanup" Nov 03 15:45:17.627001 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.626981 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" Nov 03 15:45:17.638887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.638861 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e79fa4b-f118-4d97-a393-ad65d2c46cc6-kube-api-access-29f8x\" (UniqueName: \"kubernetes.io/projected/6e79fa4b-f118-4d97-a393-ad65d2c46cc6-kube-api-access-29f8x\") pod \"6e79fa4b-f118-4d97-a393-ad65d2c46cc6\" (UID: \"6e79fa4b-f118-4d97-a393-ad65d2c46cc6\") " Nov 03 15:45:17.639050 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.638977 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e79fa4b-f118-4d97-a393-ad65d2c46cc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e79fa4b-f118-4d97-a393-ad65d2c46cc6-plugins-config-volume\") pod \"6e79fa4b-f118-4d97-a393-ad65d2c46cc6\" (UID: \"6e79fa4b-f118-4d97-a393-ad65d2c46cc6\") " Nov 03 15:45:17.639458 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.639429 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e79fa4b-f118-4d97-a393-ad65d2c46cc6-plugins-config-volume" pod "6e79fa4b-f118-4d97-a393-ad65d2c46cc6" (UID: "6e79fa4b-f118-4d97-a393-ad65d2c46cc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:17.639606 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.639584 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e79fa4b-f118-4d97-a393-ad65d2c46cc6-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:17.639661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.639612 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sddmr\" (UniqueName: \"kubernetes.io/projected/80e1db49-8fa1-40b7-ae7b-98c70dac0744-kube-api-access-sddmr\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:17.644473 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.644441 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e79fa4b-f118-4d97-a393-ad65d2c46cc6-kube-api-access-29f8x" pod "6e79fa4b-f118-4d97-a393-ad65d2c46cc6" (UID: "6e79fa4b-f118-4d97-a393-ad65d2c46cc6"). InnerVolumeSpecName "kube-api-access-29f8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:17.683423 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.681032 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:45:17.692810 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.692776 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj"] Nov 03 15:45:17.741224 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.741195 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vzh2\" (UniqueName: \"kubernetes.io/projected/218b350b-786d-4f94-a3a4-dbfb11c77e0e-kube-api-access-5vzh2\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-7scdj\" (UID: \"218b350b-786d-4f94-a3a4-dbfb11c77e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" Nov 03 15:45:17.741351 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.741319 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218b350b-786d-4f94-a3a4-dbfb11c77e0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-7scdj\" (UID: \"218b350b-786d-4f94-a3a4-dbfb11c77e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" Nov 03 15:45:17.741409 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.741356 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218b350b-786d-4f94-a3a4-dbfb11c77e0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-7scdj\" (UID: \"218b350b-786d-4f94-a3a4-dbfb11c77e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" Nov 03 15:45:17.741531 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.741511 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29f8x\" (UniqueName: \"kubernetes.io/projected/6e79fa4b-f118-4d97-a393-ad65d2c46cc6-kube-api-access-29f8x\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:17.746219 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.746169 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" Nov 03 15:45:17.842580 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.842545 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705-plugins-config-volume\") pod \"8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705\" (UID: \"8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705\") " Nov 03 15:45:17.842805 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.842633 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705-kube-api-access-kp7pf\" (UniqueName: \"kubernetes.io/projected/8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705-kube-api-access-kp7pf\") pod \"8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705\" (UID: \"8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705\") " Nov 03 15:45:17.843011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.842988 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705-plugins-config-volume" pod "8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705" (UID: "8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:17.843252 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.843225 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzh2\" (UniqueName: \"kubernetes.io/projected/218b350b-786d-4f94-a3a4-dbfb11c77e0e-kube-api-access-5vzh2\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-7scdj\" (UID: \"218b350b-786d-4f94-a3a4-dbfb11c77e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" Nov 03 15:45:17.843517 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.843357 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218b350b-786d-4f94-a3a4-dbfb11c77e0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-7scdj\" (UID: \"218b350b-786d-4f94-a3a4-dbfb11c77e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" Nov 03 15:45:17.843517 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.843402 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218b350b-786d-4f94-a3a4-dbfb11c77e0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-7scdj\" (UID: \"218b350b-786d-4f94-a3a4-dbfb11c77e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" Nov 03 15:45:17.843690 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.843670 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:17.844114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.844085 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218b350b-786d-4f94-a3a4-dbfb11c77e0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-7scdj\" (UID: \"218b350b-786d-4f94-a3a4-dbfb11c77e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" Nov 03 15:45:17.845622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.845590 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705-kube-api-access-kp7pf" pod "8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705" (UID: "8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705"). InnerVolumeSpecName "kube-api-access-kp7pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:17.846456 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.846434 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218b350b-786d-4f94-a3a4-dbfb11c77e0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-7scdj\" (UID: \"218b350b-786d-4f94-a3a4-dbfb11c77e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" Nov 03 15:45:17.908459 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.908434 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vzh2\" (UniqueName: \"kubernetes.io/projected/218b350b-786d-4f94-a3a4-dbfb11c77e0e-kube-api-access-5vzh2\") pod \"maas-api-revoke-keys-e2e-mt-5cad072f-7scdj\" (UID: \"218b350b-786d-4f94-a3a4-dbfb11c77e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" Nov 03 15:45:17.945312 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.945286 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp7pf\" (UniqueName: \"kubernetes.io/projected/8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705-kube-api-access-kp7pf\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:17.945413 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:17.945381 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" Nov 03 15:45:18.116020 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:18.115979 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7310-1a3dbd\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="828130bd-8ec4-42c3-8c11-c5ecd39b2a72" pod="opendatahub/test-curl-7310-1a3dbd" Nov 03 15:45:18.167153 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:18.167118 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7310-1a3dbd\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="828130bd-8ec4-42c3-8c11-c5ecd39b2a72" pod="opendatahub/test-curl-7310-1a3dbd" Nov 03 15:45:18.171050 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.171016 4290 generic.go:356] "Generic (PLEG): container finished" podID="8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705" containerID="67cae3c6688dca21ac410d6d4fd897a0717915c4c3623cefdadafea24f000aaf" exitCode=0 Nov 03 15:45:18.171194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.171073 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" Nov 03 15:45:18.171194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.171107 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" event={"ID":"8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705","Type":"ContainerDied","Data":"67cae3c6688dca21ac410d6d4fd897a0717915c4c3623cefdadafea24f000aaf"} Nov 03 15:45:18.171194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.171157 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h" event={"ID":"8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705","Type":"ContainerDied","Data":"f1ffb8b1c3e044483d0511bf889c60e69a062d04346b8a041b6507a10f44e136"} Nov 03 15:45:18.171194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.171186 4290 scope.go:122] "RemoveContainer" containerID="67cae3c6688dca21ac410d6d4fd897a0717915c4c3623cefdadafea24f000aaf" Nov 03 15:45:18.174083 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.173985 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" Nov 03 15:45:18.174083 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.174017 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx" event={"ID":"80e1db49-8fa1-40b7-ae7b-98c70dac0744","Type":"ContainerDied","Data":"63ea536d52dc33ba196ea17cc22e2bcf0f871f3b73daec6a12f863fbc64a0e6c"} Nov 03 15:45:18.182290 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.182264 4290 generic.go:356] "Generic (PLEG): container finished" podID="17557e9e-462a-4464-a66b-10b0f892bec5" containerID="788ab550b29c372e0ceec5b32c2089c85c3b73c68407a3c92543e216027740c8" exitCode=60 Nov 03 15:45:18.183578 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.182447 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" event={"ID":"17557e9e-462a-4464-a66b-10b0f892bec5","Type":"ContainerDied","Data":"788ab550b29c372e0ceec5b32c2089c85c3b73c68407a3c92543e216027740c8"} Nov 03 15:45:18.183578 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.182945 4290 scope.go:122] "RemoveContainer" containerID="788ab550b29c372e0ceec5b32c2089c85c3b73c68407a3c92543e216027740c8" Nov 03 15:45:18.183578 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:18.183255 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7_odh-ai-gateway-infra(17557e9e-462a-4464-a66b-10b0f892bec5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" podUID="17557e9e-462a-4464-a66b-10b0f892bec5" Nov 03 15:45:18.186376 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.186357 4290 generic.go:356] "Generic (PLEG): container finished" podID="8548a302-2c94-4f27-89c0-653e460d571a" containerID="1c89f96ced74deed85197addf437368b043a79df66ef76d342f61269bacf2e2f" exitCode=60 Nov 03 15:45:18.186490 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.186378 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" event={"ID":"8548a302-2c94-4f27-89c0-653e460d571a","Type":"ContainerDied","Data":"1c89f96ced74deed85197addf437368b043a79df66ef76d342f61269bacf2e2f"} Nov 03 15:45:18.186930 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.186909 4290 scope.go:122] "RemoveContainer" containerID="1c89f96ced74deed85197addf437368b043a79df66ef76d342f61269bacf2e2f" Nov 03 15:45:18.187254 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:18.187229 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g_odh-ai-gateway-infra(8548a302-2c94-4f27-89c0-653e460d571a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" podUID="8548a302-2c94-4f27-89c0-653e460d571a" Nov 03 15:45:18.187762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.187485 4290 scope.go:122] "RemoveContainer" containerID="67cae3c6688dca21ac410d6d4fd897a0717915c4c3623cefdadafea24f000aaf" Nov 03 15:45:18.187846 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:18.187796 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67cae3c6688dca21ac410d6d4fd897a0717915c4c3623cefdadafea24f000aaf\": container with ID starting with 67cae3c6688dca21ac410d6d4fd897a0717915c4c3623cefdadafea24f000aaf not found: ID does not exist" containerID="67cae3c6688dca21ac410d6d4fd897a0717915c4c3623cefdadafea24f000aaf" Nov 03 15:45:18.187846 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.187819 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67cae3c6688dca21ac410d6d4fd897a0717915c4c3623cefdadafea24f000aaf"} err="failed to get container status \"67cae3c6688dca21ac410d6d4fd897a0717915c4c3623cefdadafea24f000aaf\": rpc error: code = NotFound desc = could not find container \"67cae3c6688dca21ac410d6d4fd897a0717915c4c3623cefdadafea24f000aaf\": container with ID starting with 67cae3c6688dca21ac410d6d4fd897a0717915c4c3623cefdadafea24f000aaf not found: ID does not exist" Nov 03 15:45:18.187846 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.187836 4290 scope.go:122] "RemoveContainer" containerID="1ee1c80d038c1ab077f4a0d99229a7e56c985addbbb7c57859a6e71ed1cb67b4" Nov 03 15:45:18.188312 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.188290 4290 generic.go:356] "Generic (PLEG): container finished" podID="6e79fa4b-f118-4d97-a393-ad65d2c46cc6" containerID="3361c095289b146d7674f7febb333596add3af550024f98d09de932ac76383e0" exitCode=0 Nov 03 15:45:18.188379 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.188343 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" Nov 03 15:45:18.188464 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.188439 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" event={"ID":"6e79fa4b-f118-4d97-a393-ad65d2c46cc6","Type":"ContainerDied","Data":"3361c095289b146d7674f7febb333596add3af550024f98d09de932ac76383e0"} Nov 03 15:45:18.188557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.188535 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf" event={"ID":"6e79fa4b-f118-4d97-a393-ad65d2c46cc6","Type":"ContainerDied","Data":"848a48a22c67e1144ad514c5c0a03abfdf3f64efba7b4bf34b47ebd9cbf6bd37"} Nov 03 15:45:18.200421 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.200331 4290 scope.go:122] "RemoveContainer" containerID="6c6e95cc345245e85e6f7a4be50577e760ee7eb16126da4dd118d4b029e670ae" Nov 03 15:45:18.222327 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.222301 4290 scope.go:122] "RemoveContainer" containerID="612956036e8dcb40dfd70e2d08644ec57295e54b531e9b59119d9c36821400c1" Nov 03 15:45:18.240785 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.240766 4290 scope.go:122] "RemoveContainer" containerID="3361c095289b146d7674f7febb333596add3af550024f98d09de932ac76383e0" Nov 03 15:45:18.253657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.253639 4290 scope.go:122] "RemoveContainer" containerID="3361c095289b146d7674f7febb333596add3af550024f98d09de932ac76383e0" Nov 03 15:45:18.254015 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:18.253989 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3361c095289b146d7674f7febb333596add3af550024f98d09de932ac76383e0\": container with ID starting with 3361c095289b146d7674f7febb333596add3af550024f98d09de932ac76383e0 not found: ID does not exist" containerID="3361c095289b146d7674f7febb333596add3af550024f98d09de932ac76383e0" Nov 03 15:45:18.254105 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.254024 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3361c095289b146d7674f7febb333596add3af550024f98d09de932ac76383e0"} err="failed to get container status \"3361c095289b146d7674f7febb333596add3af550024f98d09de932ac76383e0\": rpc error: code = NotFound desc = could not find container \"3361c095289b146d7674f7febb333596add3af550024f98d09de932ac76383e0\": container with ID starting with 3361c095289b146d7674f7febb333596add3af550024f98d09de932ac76383e0 not found: ID does not exist" Nov 03 15:45:18.327147 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.327071 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7"] Nov 03 15:45:18.369673 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.369648 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj"] Nov 03 15:45:18.434410 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.434381 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g"] Nov 03 15:45:18.467873 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.463733 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx"] Nov 03 15:45:18.484376 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.484315 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-90fcca-29895345-4m2jx"] Nov 03 15:45:18.523868 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:18.523824 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7310-1a3dbd\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="828130bd-8ec4-42c3-8c11-c5ecd39b2a72" pod="opendatahub/test-curl-7310-1a3dbd" Nov 03 15:45:18.557335 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.557307 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm"] Nov 03 15:45:18.566791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.566763 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" Nov 03 15:45:18.647911 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.646155 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h"] Nov 03 15:45:18.665767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.658483 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be07c854-6c6c-4760-a22e-7b28af22ae21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be07c854-6c6c-4760-a22e-7b28af22ae21-maas-api-service-ca\") pod \"be07c854-6c6c-4760-a22e-7b28af22ae21\" (UID: \"be07c854-6c6c-4760-a22e-7b28af22ae21\") " Nov 03 15:45:18.665767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.658760 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be07c854-6c6c-4760-a22e-7b28af22ae21-kube-api-access-94mdx\" (UniqueName: \"kubernetes.io/projected/be07c854-6c6c-4760-a22e-7b28af22ae21-kube-api-access-94mdx\") pod \"be07c854-6c6c-4760-a22e-7b28af22ae21\" (UID: \"be07c854-6c6c-4760-a22e-7b28af22ae21\") " Nov 03 15:45:18.668549 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.668526 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-90fcca-6d54954647-zxf8h"] Nov 03 15:45:18.670128 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.670096 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be07c854-6c6c-4760-a22e-7b28af22ae21-maas-api-service-ca" pod "be07c854-6c6c-4760-a22e-7b28af22ae21" (UID: "be07c854-6c6c-4760-a22e-7b28af22ae21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:18.671551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.671526 4290 scope.go:122] "RemoveContainer" containerID="65c5a5278f6d0ba87b4f2b3ff4b34a8a30b82ecb377e64fd69a01bd574a10720" Nov 03 15:45:18.683001 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.682974 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" path="/var/lib/kubelet/pods/80e1db49-8fa1-40b7-ae7b-98c70dac0744/volumes" Nov 03 15:45:18.683415 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.683399 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705" path="/var/lib/kubelet/pods/8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705/volumes" Nov 03 15:45:18.727446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.727419 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7"] Nov 03 15:45:18.743334 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.741339 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be07c854-6c6c-4760-a22e-7b28af22ae21-kube-api-access-94mdx" pod "be07c854-6c6c-4760-a22e-7b28af22ae21" (UID: "be07c854-6c6c-4760-a22e-7b28af22ae21"). InnerVolumeSpecName "kube-api-access-94mdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:18.769171 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.768346 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94mdx\" (UniqueName: \"kubernetes.io/projected/be07c854-6c6c-4760-a22e-7b28af22ae21-kube-api-access-94mdx\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:18.769822 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.769800 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be07c854-6c6c-4760-a22e-7b28af22ae21-maas-api-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:18.946102 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:18.945986 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" Nov 03 15:45:19.034500 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.034467 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf"] Nov 03 15:45:19.042466 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.042436 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-90fcca-b997c7df-6xctf"] Nov 03 15:45:19.073864 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.073824 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2094a7-5e86-43c9-8e0c-917426278359-kube-api-access-jpkp4\" (UniqueName: \"kubernetes.io/projected/7d2094a7-5e86-43c9-8e0c-917426278359-kube-api-access-jpkp4\") pod \"7d2094a7-5e86-43c9-8e0c-917426278359\" (UID: \"7d2094a7-5e86-43c9-8e0c-917426278359\") " Nov 03 15:45:19.074020 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.074005 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2094a7-5e86-43c9-8e0c-917426278359-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2094a7-5e86-43c9-8e0c-917426278359-maas-api-service-ca\") pod \"7d2094a7-5e86-43c9-8e0c-917426278359\" (UID: \"7d2094a7-5e86-43c9-8e0c-917426278359\") " Nov 03 15:45:19.074369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.074337 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2094a7-5e86-43c9-8e0c-917426278359-maas-api-service-ca" pod "7d2094a7-5e86-43c9-8e0c-917426278359" (UID: "7d2094a7-5e86-43c9-8e0c-917426278359"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:19.074688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.074653 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2094a7-5e86-43c9-8e0c-917426278359-maas-api-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:19.076416 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.076389 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2094a7-5e86-43c9-8e0c-917426278359-kube-api-access-jpkp4" pod "7d2094a7-5e86-43c9-8e0c-917426278359" (UID: "7d2094a7-5e86-43c9-8e0c-917426278359"). InnerVolumeSpecName "kube-api-access-jpkp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:19.176949 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.176914 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpkp4\" (UniqueName: \"kubernetes.io/projected/7d2094a7-5e86-43c9-8e0c-917426278359-kube-api-access-jpkp4\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:19.207216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.207190 4290 generic.go:356] "Generic (PLEG): container finished" podID="828130bd-8ec4-42c3-8c11-c5ecd39b2a72" containerID="50967f62ca308591b09442d6c5c6b38b7583ffc8d7146a457d9574195728c0d5" exitCode=137 Nov 03 15:45:19.212081 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.212054 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" event={"ID":"321f6b11-c456-4a61-9649-b3a37398086f","Type":"ContainerStarted","Data":"3375a7fb290b1c7924f8ef7c94a409e10ea203257ee02734209ee851bb745ab4"} Nov 03 15:45:19.217782 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.214981 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" event={"ID":"be07c854-6c6c-4760-a22e-7b28af22ae21","Type":"ContainerDied","Data":"501e033a6c306669e377b93152f34470653bd4a043dc19547ddcd49c85920510"} Nov 03 15:45:19.217782 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.215013 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7" Nov 03 15:45:19.217782 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.215038 4290 scope.go:122] "RemoveContainer" containerID="f182e2fe4d8d6bcf76b46e98e9845b882f002777bda846f5e11d69254ddc5ced" Nov 03 15:45:19.223192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.222455 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" event={"ID":"218b350b-786d-4f94-a3a4-dbfb11c77e0e","Type":"ContainerStarted","Data":"bdfc1dea020544ae5174ebdb4546169b618e02154fbe486bceae7958568d9350"} Nov 03 15:45:19.223192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.222848 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" event={"ID":"218b350b-786d-4f94-a3a4-dbfb11c77e0e","Type":"ContainerStarted","Data":"3738eb72a1ffdb0b13a7f338b747eb3297832af85d55544ab9cd02041ae4f572"} Nov 03 15:45:19.227081 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.226096 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" event={"ID":"7d2094a7-5e86-43c9-8e0c-917426278359","Type":"ContainerDied","Data":"a4e7f2f94401cc5d14b48b6bd41a4603fef8088ae2f370a2e195b7af0d0338f8"} Nov 03 15:45:19.227081 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.226172 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm" Nov 03 15:45:19.235480 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.235442 4290 scope.go:122] "RemoveContainer" containerID="05f4798973b46eff4e563242ac6a0a1a0cbd31e5c2587617af8f07193ed9ea1c" Nov 03 15:45:19.501491 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.501392 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" Nov 03 15:45:19.517793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.517663 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" Nov 03 15:45:19.569781 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.567381 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm"] Nov 03 15:45:19.572530 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.572503 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7310-1a3dbd" Nov 03 15:45:19.577927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.577891 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fe0cb9-29895345-x52mm"] Nov 03 15:45:19.583114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.583091 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17557e9e-462a-4464-a66b-10b0f892bec5-kube-api-access-bhfrs\" (UniqueName: \"kubernetes.io/projected/17557e9e-462a-4464-a66b-10b0f892bec5-kube-api-access-bhfrs\") pod \"17557e9e-462a-4464-a66b-10b0f892bec5\" (UID: \"17557e9e-462a-4464-a66b-10b0f892bec5\") " Nov 03 15:45:19.583243 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.583226 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17557e9e-462a-4464-a66b-10b0f892bec5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17557e9e-462a-4464-a66b-10b0f892bec5-maas-api-service-ca\") pod \"17557e9e-462a-4464-a66b-10b0f892bec5\" (UID: \"17557e9e-462a-4464-a66b-10b0f892bec5\") " Nov 03 15:45:19.583324 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.583245 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8548a302-2c94-4f27-89c0-653e460d571a-kube-api-access-4fn8b\" (UniqueName: \"kubernetes.io/projected/8548a302-2c94-4f27-89c0-653e460d571a-kube-api-access-4fn8b\") pod \"8548a302-2c94-4f27-89c0-653e460d571a\" (UID: \"8548a302-2c94-4f27-89c0-653e460d571a\") " Nov 03 15:45:19.583382 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.583365 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8548a302-2c94-4f27-89c0-653e460d571a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8548a302-2c94-4f27-89c0-653e460d571a-maas-api-service-ca\") pod \"8548a302-2c94-4f27-89c0-653e460d571a\" (UID: \"8548a302-2c94-4f27-89c0-653e460d571a\") " Nov 03 15:45:19.583915 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.583888 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17557e9e-462a-4464-a66b-10b0f892bec5-maas-api-service-ca" pod "17557e9e-462a-4464-a66b-10b0f892bec5" (UID: "17557e9e-462a-4464-a66b-10b0f892bec5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:19.584224 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.584201 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8548a302-2c94-4f27-89c0-653e460d571a-maas-api-service-ca" pod "8548a302-2c94-4f27-89c0-653e460d571a" (UID: "8548a302-2c94-4f27-89c0-653e460d571a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:19.584457 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.584372 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8548a302-2c94-4f27-89c0-653e460d571a-maas-api-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:19.584457 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.584389 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17557e9e-462a-4464-a66b-10b0f892bec5-maas-api-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:19.586024 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.586003 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17557e9e-462a-4464-a66b-10b0f892bec5-kube-api-access-bhfrs" pod "17557e9e-462a-4464-a66b-10b0f892bec5" (UID: "17557e9e-462a-4464-a66b-10b0f892bec5"). InnerVolumeSpecName "kube-api-access-bhfrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:19.586103 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.586031 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8548a302-2c94-4f27-89c0-653e460d571a-kube-api-access-4fn8b" pod "8548a302-2c94-4f27-89c0-653e460d571a" (UID: "8548a302-2c94-4f27-89c0-653e460d571a"). InnerVolumeSpecName "kube-api-access-4fn8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:19.685889 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.685855 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828130bd-8ec4-42c3-8c11-c5ecd39b2a72-kube-api-access-8v4np\" (UniqueName: \"kubernetes.io/projected/828130bd-8ec4-42c3-8c11-c5ecd39b2a72-kube-api-access-8v4np\") pod \"828130bd-8ec4-42c3-8c11-c5ecd39b2a72\" (UID: \"828130bd-8ec4-42c3-8c11-c5ecd39b2a72\") " Nov 03 15:45:19.686622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.686565 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fn8b\" (UniqueName: \"kubernetes.io/projected/8548a302-2c94-4f27-89c0-653e460d571a-kube-api-access-4fn8b\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:19.686622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.686591 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhfrs\" (UniqueName: \"kubernetes.io/projected/17557e9e-462a-4464-a66b-10b0f892bec5-kube-api-access-bhfrs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:19.688122 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.688095 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828130bd-8ec4-42c3-8c11-c5ecd39b2a72-kube-api-access-8v4np" pod "828130bd-8ec4-42c3-8c11-c5ecd39b2a72" (UID: "828130bd-8ec4-42c3-8c11-c5ecd39b2a72"). InnerVolumeSpecName "kube-api-access-8v4np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:19.788579 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.788485 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v4np\" (UniqueName: \"kubernetes.io/projected/828130bd-8ec4-42c3-8c11-c5ecd39b2a72-kube-api-access-8v4np\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:19.794248 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.794220 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7"] Nov 03 15:45:19.798896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.798872 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-38ded0-29895345-pcbg7"] Nov 03 15:45:19.918834 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:19.918770 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" podStartSLOduration=2.918749317 podStartE2EDuration="2.918749317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:19.909991395 +0000 UTC m=+2227.340949225" watchObservedRunningTime="2026-11-03 15:45:19.918749317 +0000 UTC m=+2227.349707127" Nov 03 15:45:20.241201 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.241166 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" event={"ID":"8548a302-2c94-4f27-89c0-653e460d571a","Type":"ContainerDied","Data":"10ba5411e00fba814feeffb8735ce83548f738c4922a1a509df87d5ceaab0de1"} Nov 03 15:45:20.244080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.241237 4290 scope.go:122] "RemoveContainer" containerID="1c89f96ced74deed85197addf437368b043a79df66ef76d342f61269bacf2e2f" Nov 03 15:45:20.244080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.241172 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g" Nov 03 15:45:20.244080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.243275 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7310-1a3dbd" Nov 03 15:45:20.246248 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.246213 4290 generic.go:356] "Generic (PLEG): container finished" podID="321f6b11-c456-4a61-9649-b3a37398086f" containerID="3375a7fb290b1c7924f8ef7c94a409e10ea203257ee02734209ee851bb745ab4" exitCode=60 Nov 03 15:45:20.246375 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.246316 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" event={"ID":"321f6b11-c456-4a61-9649-b3a37398086f","Type":"ContainerDied","Data":"3375a7fb290b1c7924f8ef7c94a409e10ea203257ee02734209ee851bb745ab4"} Nov 03 15:45:20.247091 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.247069 4290 scope.go:122] "RemoveContainer" containerID="3375a7fb290b1c7924f8ef7c94a409e10ea203257ee02734209ee851bb745ab4" Nov 03 15:45:20.247439 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:20.247417 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw_odh-ai-gateway-infra(321f6b11-c456-4a61-9649-b3a37398086f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" podUID="321f6b11-c456-4a61-9649-b3a37398086f" Nov 03 15:45:20.254073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.254051 4290 generic.go:356] "Generic (PLEG): container finished" podID="218b350b-786d-4f94-a3a4-dbfb11c77e0e" containerID="bdfc1dea020544ae5174ebdb4546169b618e02154fbe486bceae7958568d9350" exitCode=0 Nov 03 15:45:20.254177 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.254143 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" event={"ID":"218b350b-786d-4f94-a3a4-dbfb11c77e0e","Type":"ContainerDied","Data":"bdfc1dea020544ae5174ebdb4546169b618e02154fbe486bceae7958568d9350"} Nov 03 15:45:20.264985 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.262312 4290 scope.go:122] "RemoveContainer" containerID="50967f62ca308591b09442d6c5c6b38b7583ffc8d7146a457d9574195728c0d5" Nov 03 15:45:20.266737 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.266691 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" event={"ID":"17557e9e-462a-4464-a66b-10b0f892bec5","Type":"ContainerDied","Data":"186e93ecbedd406154f39d7387ad08f31c9d8a394a63d3166bc3094248988f0f"} Nov 03 15:45:20.266892 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.266808 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7" Nov 03 15:45:20.276185 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.276161 4290 scope.go:122] "RemoveContainer" containerID="65c5a5278f6d0ba87b4f2b3ff4b34a8a30b82ecb377e64fd69a01bd574a10720" Nov 03 15:45:20.291230 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.291206 4290 scope.go:122] "RemoveContainer" containerID="788ab550b29c372e0ceec5b32c2089c85c3b73c68407a3c92543e216027740c8" Nov 03 15:45:20.608230 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.608149 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g"] Nov 03 15:45:20.613024 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.612998 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-75edeb-29895345-lmk2g"] Nov 03 15:45:20.688240 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.688201 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e79fa4b-f118-4d97-a393-ad65d2c46cc6" path="/var/lib/kubelet/pods/6e79fa4b-f118-4d97-a393-ad65d2c46cc6/volumes" Nov 03 15:45:20.689191 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.689165 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2094a7-5e86-43c9-8e0c-917426278359" path="/var/lib/kubelet/pods/7d2094a7-5e86-43c9-8e0c-917426278359/volumes" Nov 03 15:45:20.689972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.689952 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="828130bd-8ec4-42c3-8c11-c5ecd39b2a72" path="/var/lib/kubelet/pods/828130bd-8ec4-42c3-8c11-c5ecd39b2a72/volumes" Nov 03 15:45:20.691155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.691136 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8548a302-2c94-4f27-89c0-653e460d571a" path="/var/lib/kubelet/pods/8548a302-2c94-4f27-89c0-653e460d571a/volumes" Nov 03 15:45:20.692217 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.692187 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be07c854-6c6c-4760-a22e-7b28af22ae21" path="/var/lib/kubelet/pods/be07c854-6c6c-4760-a22e-7b28af22ae21/volumes" Nov 03 15:45:20.692794 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.692778 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw"] Nov 03 15:45:20.722502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.722473 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7"] Nov 03 15:45:20.727271 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.727245 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5cad072f-29895345-8clq7"] Nov 03 15:45:20.749304 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.749276 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn"] Nov 03 15:45:20.754345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:20.754317 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-38ded0-g9mdn"] Nov 03 15:45:20.772827 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:20.772793 4290 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7310-1a3dbd\" is forbidden: User \"system:node:ip-10-0-139-35.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-35.us-west-1.compute.internal' and this object" podUID="828130bd-8ec4-42c3-8c11-c5ecd39b2a72" pod="opendatahub/test-curl-7310-1a3dbd" Nov 03 15:45:21.530479 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.530453 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" Nov 03 15:45:21.533229 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.533179 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" Nov 03 15:45:21.617308 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.617280 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218b350b-786d-4f94-a3a4-dbfb11c77e0e-kube-api-access-5vzh2\" (UniqueName: \"kubernetes.io/projected/218b350b-786d-4f94-a3a4-dbfb11c77e0e-kube-api-access-5vzh2\") pod \"218b350b-786d-4f94-a3a4-dbfb11c77e0e\" (UID: \"218b350b-786d-4f94-a3a4-dbfb11c77e0e\") " Nov 03 15:45:21.617468 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.617323 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321f6b11-c456-4a61-9649-b3a37398086f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/321f6b11-c456-4a61-9649-b3a37398086f-maas-api-service-ca\") pod \"321f6b11-c456-4a61-9649-b3a37398086f\" (UID: \"321f6b11-c456-4a61-9649-b3a37398086f\") " Nov 03 15:45:21.617468 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.617445 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/218b350b-786d-4f94-a3a4-dbfb11c77e0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218b350b-786d-4f94-a3a4-dbfb11c77e0e-maas-api-xks-ca\") pod \"218b350b-786d-4f94-a3a4-dbfb11c77e0e\" (UID: \"218b350b-786d-4f94-a3a4-dbfb11c77e0e\") " Nov 03 15:45:21.617606 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.617587 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321f6b11-c456-4a61-9649-b3a37398086f-kube-api-access-czbkz\" (UniqueName: \"kubernetes.io/projected/321f6b11-c456-4a61-9649-b3a37398086f-kube-api-access-czbkz\") pod \"321f6b11-c456-4a61-9649-b3a37398086f\" (UID: \"321f6b11-c456-4a61-9649-b3a37398086f\") " Nov 03 15:45:21.617812 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.617787 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321f6b11-c456-4a61-9649-b3a37398086f-maas-api-service-ca" pod "321f6b11-c456-4a61-9649-b3a37398086f" (UID: "321f6b11-c456-4a61-9649-b3a37398086f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:21.617916 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.617829 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/218b350b-786d-4f94-a3a4-dbfb11c77e0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218b350b-786d-4f94-a3a4-dbfb11c77e0e-maas-api-openshift-ca\") pod \"218b350b-786d-4f94-a3a4-dbfb11c77e0e\" (UID: \"218b350b-786d-4f94-a3a4-dbfb11c77e0e\") " Nov 03 15:45:21.618410 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.618353 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218b350b-786d-4f94-a3a4-dbfb11c77e0e-maas-api-openshift-ca" pod "218b350b-786d-4f94-a3a4-dbfb11c77e0e" (UID: "218b350b-786d-4f94-a3a4-dbfb11c77e0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:21.618685 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.618663 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218b350b-786d-4f94-a3a4-dbfb11c77e0e-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:21.618856 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.618691 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/321f6b11-c456-4a61-9649-b3a37398086f-maas-api-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:21.619933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.619907 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218b350b-786d-4f94-a3a4-dbfb11c77e0e-maas-api-xks-ca" pod "218b350b-786d-4f94-a3a4-dbfb11c77e0e" (UID: "218b350b-786d-4f94-a3a4-dbfb11c77e0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:45:21.620013 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.619969 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218b350b-786d-4f94-a3a4-dbfb11c77e0e-kube-api-access-5vzh2" pod "218b350b-786d-4f94-a3a4-dbfb11c77e0e" (UID: "218b350b-786d-4f94-a3a4-dbfb11c77e0e"). InnerVolumeSpecName "kube-api-access-5vzh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:21.620061 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.620017 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321f6b11-c456-4a61-9649-b3a37398086f-kube-api-access-czbkz" pod "321f6b11-c456-4a61-9649-b3a37398086f" (UID: "321f6b11-c456-4a61-9649-b3a37398086f"). InnerVolumeSpecName "kube-api-access-czbkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:21.719991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.719958 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vzh2\" (UniqueName: \"kubernetes.io/projected/218b350b-786d-4f94-a3a4-dbfb11c77e0e-kube-api-access-5vzh2\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:21.719991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.719988 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218b350b-786d-4f94-a3a4-dbfb11c77e0e-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:21.719991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:21.719997 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czbkz\" (UniqueName: \"kubernetes.io/projected/321f6b11-c456-4a61-9649-b3a37398086f-kube-api-access-czbkz\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:22.293537 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:22.293503 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" Nov 03 15:45:22.293774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:22.293515 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw" event={"ID":"321f6b11-c456-4a61-9649-b3a37398086f","Type":"ContainerDied","Data":"c7bb3769f2d7a0475f056895f53c203697e889e061df04e69603463433b2252c"} Nov 03 15:45:22.293774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:22.293659 4290 scope.go:122] "RemoveContainer" containerID="3375a7fb290b1c7924f8ef7c94a409e10ea203257ee02734209ee851bb745ab4" Nov 03 15:45:22.295418 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:22.295391 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" Nov 03 15:45:22.295520 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:22.295432 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj" event={"ID":"218b350b-786d-4f94-a3a4-dbfb11c77e0e","Type":"ContainerDied","Data":"3738eb72a1ffdb0b13a7f338b747eb3297832af85d55544ab9cd02041ae4f572"} Nov 03 15:45:22.295520 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:22.295485 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3738eb72a1ffdb0b13a7f338b747eb3297832af85d55544ab9cd02041ae4f572" Nov 03 15:45:22.463530 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:22.463497 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw"] Nov 03 15:45:22.511814 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:22.511776 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-78034f-29895345-vvjsw"] Nov 03 15:45:22.679602 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:22.679573 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ca101a0-06bb-4911-997f-be22cb2a1574" path="/var/lib/kubelet/pods/0ca101a0-06bb-4911-997f-be22cb2a1574/volumes" Nov 03 15:45:22.679994 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:22.679980 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17557e9e-462a-4464-a66b-10b0f892bec5" path="/var/lib/kubelet/pods/17557e9e-462a-4464-a66b-10b0f892bec5/volumes" Nov 03 15:45:22.680343 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:22.680330 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321f6b11-c456-4a61-9649-b3a37398086f" path="/var/lib/kubelet/pods/321f6b11-c456-4a61-9649-b3a37398086f/volumes" Nov 03 15:45:30.482543 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:30.482508 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z"] Nov 03 15:45:30.594506 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:30.594475 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6"] Nov 03 15:45:35.483062 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:35.483015 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" podUID="60c5ffbd-5f53-447d-b85b-538a4f326c7f" containerName="payload-processing" containerID="cri-o://57b52786c7bb5a61e90615777968eabe0cea1a0fb99bb0c43aa0336ad50689c1" gracePeriod=25 Nov 03 15:45:35.595775 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:35.595735 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" podUID="1b6b120a-3b26-4f20-a145-ccdc060e6ea8" containerName="payload-pre-processing" containerID="cri-o://f97e37a45ef5f6ad9505165cfc538b7183c7086e8cffd360092c5201307713c5" gracePeriod=25 Nov 03 15:45:35.826981 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:35.826951 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" Nov 03 15:45:35.960369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:35.960337 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" Nov 03 15:45:35.972038 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:35.972003 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c5ffbd-5f53-447d-b85b-538a4f326c7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c5ffbd-5f53-447d-b85b-538a4f326c7f-plugins-config-volume\") pod \"60c5ffbd-5f53-447d-b85b-538a4f326c7f\" (UID: \"60c5ffbd-5f53-447d-b85b-538a4f326c7f\") " Nov 03 15:45:35.972182 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:35.972090 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c5ffbd-5f53-447d-b85b-538a4f326c7f-kube-api-access-xkj96\" (UniqueName: \"kubernetes.io/projected/60c5ffbd-5f53-447d-b85b-538a4f326c7f-kube-api-access-xkj96\") pod \"60c5ffbd-5f53-447d-b85b-538a4f326c7f\" (UID: \"60c5ffbd-5f53-447d-b85b-538a4f326c7f\") " Nov 03 15:45:35.972447 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:35.972422 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c5ffbd-5f53-447d-b85b-538a4f326c7f-plugins-config-volume" pod "60c5ffbd-5f53-447d-b85b-538a4f326c7f" (UID: "60c5ffbd-5f53-447d-b85b-538a4f326c7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:35.972738 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:35.972717 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c5ffbd-5f53-447d-b85b-538a4f326c7f-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:35.975018 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:35.974989 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c5ffbd-5f53-447d-b85b-538a4f326c7f-kube-api-access-xkj96" pod "60c5ffbd-5f53-447d-b85b-538a4f326c7f" (UID: "60c5ffbd-5f53-447d-b85b-538a4f326c7f"). InnerVolumeSpecName "kube-api-access-xkj96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:36.074263 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.074169 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b6b120a-3b26-4f20-a145-ccdc060e6ea8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6b120a-3b26-4f20-a145-ccdc060e6ea8-plugins-config-volume\") pod \"1b6b120a-3b26-4f20-a145-ccdc060e6ea8\" (UID: \"1b6b120a-3b26-4f20-a145-ccdc060e6ea8\") " Nov 03 15:45:36.074433 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.074386 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6b120a-3b26-4f20-a145-ccdc060e6ea8-kube-api-access-wvzjw\" (UniqueName: \"kubernetes.io/projected/1b6b120a-3b26-4f20-a145-ccdc060e6ea8-kube-api-access-wvzjw\") pod \"1b6b120a-3b26-4f20-a145-ccdc060e6ea8\" (UID: \"1b6b120a-3b26-4f20-a145-ccdc060e6ea8\") " Nov 03 15:45:36.074562 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.074536 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b6b120a-3b26-4f20-a145-ccdc060e6ea8-plugins-config-volume" pod "1b6b120a-3b26-4f20-a145-ccdc060e6ea8" (UID: "1b6b120a-3b26-4f20-a145-ccdc060e6ea8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:36.074902 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.074886 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkj96\" (UniqueName: \"kubernetes.io/projected/60c5ffbd-5f53-447d-b85b-538a4f326c7f-kube-api-access-xkj96\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:36.074975 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.074905 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6b120a-3b26-4f20-a145-ccdc060e6ea8-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:36.076685 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.076656 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6b120a-3b26-4f20-a145-ccdc060e6ea8-kube-api-access-wvzjw" pod "1b6b120a-3b26-4f20-a145-ccdc060e6ea8" (UID: "1b6b120a-3b26-4f20-a145-ccdc060e6ea8"). InnerVolumeSpecName "kube-api-access-wvzjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:36.177060 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.177021 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvzjw\" (UniqueName: \"kubernetes.io/projected/1b6b120a-3b26-4f20-a145-ccdc060e6ea8-kube-api-access-wvzjw\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:36.479138 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.479089 4290 generic.go:356] "Generic (PLEG): container finished" podID="60c5ffbd-5f53-447d-b85b-538a4f326c7f" containerID="57b52786c7bb5a61e90615777968eabe0cea1a0fb99bb0c43aa0336ad50689c1" exitCode=0 Nov 03 15:45:36.479796 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.479747 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" event={"ID":"60c5ffbd-5f53-447d-b85b-538a4f326c7f","Type":"ContainerDied","Data":"57b52786c7bb5a61e90615777968eabe0cea1a0fb99bb0c43aa0336ad50689c1"} Nov 03 15:45:36.480034 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.479998 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" event={"ID":"60c5ffbd-5f53-447d-b85b-538a4f326c7f","Type":"ContainerDied","Data":"0393f3ab73cd10592603ae50632766bf027eb528e09763890b919c89f84af109"} Nov 03 15:45:36.480243 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.480179 4290 scope.go:122] "RemoveContainer" containerID="57b52786c7bb5a61e90615777968eabe0cea1a0fb99bb0c43aa0336ad50689c1" Nov 03 15:45:36.480634 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.480617 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z" Nov 03 15:45:36.526032 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.516792 4290 generic.go:356] "Generic (PLEG): container finished" podID="1b6b120a-3b26-4f20-a145-ccdc060e6ea8" containerID="f97e37a45ef5f6ad9505165cfc538b7183c7086e8cffd360092c5201307713c5" exitCode=0 Nov 03 15:45:36.526032 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.516969 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" Nov 03 15:45:36.526032 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.517086 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" event={"ID":"1b6b120a-3b26-4f20-a145-ccdc060e6ea8","Type":"ContainerDied","Data":"f97e37a45ef5f6ad9505165cfc538b7183c7086e8cffd360092c5201307713c5"} Nov 03 15:45:36.526032 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.517126 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6" event={"ID":"1b6b120a-3b26-4f20-a145-ccdc060e6ea8","Type":"ContainerDied","Data":"3074332888f9b784fa6a66a308baed7f10965b6de5a8c75514dca73d93aa8d8e"} Nov 03 15:45:36.541948 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.539809 4290 scope.go:122] "RemoveContainer" containerID="57b52786c7bb5a61e90615777968eabe0cea1a0fb99bb0c43aa0336ad50689c1" Nov 03 15:45:36.541948 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:36.540399 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b52786c7bb5a61e90615777968eabe0cea1a0fb99bb0c43aa0336ad50689c1\": container with ID starting with 57b52786c7bb5a61e90615777968eabe0cea1a0fb99bb0c43aa0336ad50689c1 not found: ID does not exist" containerID="57b52786c7bb5a61e90615777968eabe0cea1a0fb99bb0c43aa0336ad50689c1" Nov 03 15:45:36.541948 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.540441 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b52786c7bb5a61e90615777968eabe0cea1a0fb99bb0c43aa0336ad50689c1"} err="failed to get container status \"57b52786c7bb5a61e90615777968eabe0cea1a0fb99bb0c43aa0336ad50689c1\": rpc error: code = NotFound desc = could not find container \"57b52786c7bb5a61e90615777968eabe0cea1a0fb99bb0c43aa0336ad50689c1\": container with ID starting with 57b52786c7bb5a61e90615777968eabe0cea1a0fb99bb0c43aa0336ad50689c1 not found: ID does not exist" Nov 03 15:45:36.541948 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.540461 4290 scope.go:122] "RemoveContainer" containerID="f97e37a45ef5f6ad9505165cfc538b7183c7086e8cffd360092c5201307713c5" Nov 03 15:45:36.560913 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.560890 4290 scope.go:122] "RemoveContainer" containerID="f97e37a45ef5f6ad9505165cfc538b7183c7086e8cffd360092c5201307713c5" Nov 03 15:45:36.561263 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:36.561232 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97e37a45ef5f6ad9505165cfc538b7183c7086e8cffd360092c5201307713c5\": container with ID starting with f97e37a45ef5f6ad9505165cfc538b7183c7086e8cffd360092c5201307713c5 not found: ID does not exist" containerID="f97e37a45ef5f6ad9505165cfc538b7183c7086e8cffd360092c5201307713c5" Nov 03 15:45:36.561378 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.561271 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97e37a45ef5f6ad9505165cfc538b7183c7086e8cffd360092c5201307713c5"} err="failed to get container status \"f97e37a45ef5f6ad9505165cfc538b7183c7086e8cffd360092c5201307713c5\": rpc error: code = NotFound desc = could not find container \"f97e37a45ef5f6ad9505165cfc538b7183c7086e8cffd360092c5201307713c5\": container with ID starting with f97e37a45ef5f6ad9505165cfc538b7183c7086e8cffd360092c5201307713c5 not found: ID does not exist" Nov 03 15:45:36.669026 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.668994 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6"] Nov 03 15:45:36.687769 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.687734 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-r2dk6"] Nov 03 15:45:36.786871 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.786778 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z"] Nov 03 15:45:36.787591 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:36.787570 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5cad072f-86d65986f5-csp8z"] Nov 03 15:45:37.756739 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:37.756687 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5"] Nov 03 15:45:37.806003 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:37.805972 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-75edeb-pj4t5"] Nov 03 15:45:38.681326 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:38.681291 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b6b120a-3b26-4f20-a145-ccdc060e6ea8" path="/var/lib/kubelet/pods/1b6b120a-3b26-4f20-a145-ccdc060e6ea8/volumes" Nov 03 15:45:38.681959 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:38.681941 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c5ffbd-5f53-447d-b85b-538a4f326c7f" path="/var/lib/kubelet/pods/60c5ffbd-5f53-447d-b85b-538a4f326c7f/volumes" Nov 03 15:45:38.682533 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:38.682516 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="babcf474-65c1-4912-b599-ca9c31e24276" path="/var/lib/kubelet/pods/babcf474-65c1-4912-b599-ca9c31e24276/volumes" Nov 03 15:45:38.711784 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:38.705952 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:45:38.958977 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:38.958900 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4"] Nov 03 15:45:39.008561 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:39.008527 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fe0cb9-zgqq4"] Nov 03 15:45:40.683799 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:40.683764 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8115d771-05bd-4b84-bc55-2817cf4b2fa4" path="/var/lib/kubelet/pods/8115d771-05bd-4b84-bc55-2817cf4b2fa4/volumes" Nov 03 15:45:48.198288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.198250 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x"] Nov 03 15:45:48.199183 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199160 4290 cpu_manager.go:414] "containerMap: removing container" podUID="be07c854-6c6c-4760-a22e-7b28af22ae21" containerName="cleanup" Nov 03 15:45:48.199295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199184 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be07c854-6c6c-4760-a22e-7b28af22ae21" containerName="cleanup" Nov 03 15:45:48.199295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199207 4290 cpu_manager.go:414] "containerMap: removing container" podUID="218b350b-786d-4f94-a3a4-dbfb11c77e0e" containerName="revoke-keys" Nov 03 15:45:48.199295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199214 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218b350b-786d-4f94-a3a4-dbfb11c77e0e" containerName="revoke-keys" Nov 03 15:45:48.199295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199226 4290 cpu_manager.go:414] "containerMap: removing container" podUID="321f6b11-c456-4a61-9649-b3a37398086f" containerName="cleanup" Nov 03 15:45:48.199295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199233 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321f6b11-c456-4a61-9649-b3a37398086f" containerName="cleanup" Nov 03 15:45:48.199295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199245 4290 cpu_manager.go:414] "containerMap: removing container" podUID="17557e9e-462a-4464-a66b-10b0f892bec5" containerName="cleanup" Nov 03 15:45:48.199295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199253 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17557e9e-462a-4464-a66b-10b0f892bec5" containerName="cleanup" Nov 03 15:45:48.199295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199265 4290 cpu_manager.go:414] "containerMap: removing container" podUID="8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705" containerName="payload-pre-processing" Nov 03 15:45:48.199295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199274 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705" containerName="payload-pre-processing" Nov 03 15:45:48.199295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199285 4290 cpu_manager.go:414] "containerMap: removing container" podUID="17557e9e-462a-4464-a66b-10b0f892bec5" containerName="cleanup" Nov 03 15:45:48.199295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199291 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17557e9e-462a-4464-a66b-10b0f892bec5" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199306 4290 cpu_manager.go:414] "containerMap: removing container" podUID="7d2094a7-5e86-43c9-8e0c-917426278359" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199313 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2094a7-5e86-43c9-8e0c-917426278359" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199328 4290 cpu_manager.go:414] "containerMap: removing container" podUID="8548a302-2c94-4f27-89c0-653e460d571a" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199335 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8548a302-2c94-4f27-89c0-653e460d571a" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199349 4290 cpu_manager.go:414] "containerMap: removing container" podUID="17557e9e-462a-4464-a66b-10b0f892bec5" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199356 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17557e9e-462a-4464-a66b-10b0f892bec5" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199373 4290 cpu_manager.go:414] "containerMap: removing container" podUID="8548a302-2c94-4f27-89c0-653e460d571a" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199383 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8548a302-2c94-4f27-89c0-653e460d571a" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199394 4290 cpu_manager.go:414] "containerMap: removing container" podUID="7d2094a7-5e86-43c9-8e0c-917426278359" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199402 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2094a7-5e86-43c9-8e0c-917426278359" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199412 4290 cpu_manager.go:414] "containerMap: removing container" podUID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199420 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199440 4290 cpu_manager.go:414] "containerMap: removing container" podUID="321f6b11-c456-4a61-9649-b3a37398086f" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199447 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321f6b11-c456-4a61-9649-b3a37398086f" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199458 4290 cpu_manager.go:414] "containerMap: removing container" podUID="be07c854-6c6c-4760-a22e-7b28af22ae21" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199465 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be07c854-6c6c-4760-a22e-7b28af22ae21" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199473 4290 cpu_manager.go:414] "containerMap: removing container" podUID="7d2094a7-5e86-43c9-8e0c-917426278359" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199480 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2094a7-5e86-43c9-8e0c-917426278359" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199525 4290 cpu_manager.go:414] "containerMap: removing container" podUID="1b6b120a-3b26-4f20-a145-ccdc060e6ea8" containerName="payload-pre-processing" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199534 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6b120a-3b26-4f20-a145-ccdc060e6ea8" containerName="payload-pre-processing" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199551 4290 cpu_manager.go:414] "containerMap: removing container" podUID="60c5ffbd-5f53-447d-b85b-538a4f326c7f" containerName="payload-processing" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199559 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c5ffbd-5f53-447d-b85b-538a4f326c7f" containerName="payload-processing" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199572 4290 cpu_manager.go:414] "containerMap: removing container" podUID="321f6b11-c456-4a61-9649-b3a37398086f" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199580 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321f6b11-c456-4a61-9649-b3a37398086f" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199783 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="17557e9e-462a-4464-a66b-10b0f892bec5" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199796 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="be07c854-6c6c-4760-a22e-7b28af22ae21" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199806 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="8548a302-2c94-4f27-89c0-653e460d571a" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199817 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2094a7-5e86-43c9-8e0c-917426278359" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199827 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e1db49-8fa1-40b7-ae7b-98c70dac0744" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199840 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="321f6b11-c456-4a61-9649-b3a37398086f" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199847 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="be07c854-6c6c-4760-a22e-7b28af22ae21" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199856 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2094a7-5e86-43c9-8e0c-917426278359" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199869 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b6b120a-3b26-4f20-a145-ccdc060e6ea8" containerName="payload-pre-processing" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199880 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c5ffbd-5f53-447d-b85b-538a4f326c7f" containerName="payload-processing" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199891 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="321f6b11-c456-4a61-9649-b3a37398086f" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199903 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="be07c854-6c6c-4760-a22e-7b28af22ae21" containerName="cleanup" Nov 03 15:45:48.199896 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199913 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="218b350b-786d-4f94-a3a4-dbfb11c77e0e" containerName="revoke-keys" Nov 03 15:45:48.201630 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199923 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="17557e9e-462a-4464-a66b-10b0f892bec5" containerName="cleanup" Nov 03 15:45:48.201630 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199931 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7a66a2-fd0c-4f32-9c24-c8e5a9b73705" containerName="payload-pre-processing" Nov 03 15:45:48.201630 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199942 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="17557e9e-462a-4464-a66b-10b0f892bec5" containerName="cleanup" Nov 03 15:45:48.201630 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.199952 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="8548a302-2c94-4f27-89c0-653e460d571a" containerName="cleanup" Nov 03 15:45:48.205557 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.205537 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" Nov 03 15:45:48.262595 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.262558 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x"] Nov 03 15:45:48.296769 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.296690 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grx2k\" (UniqueName: \"kubernetes.io/projected/92c25563-47d0-403d-afbc-0461e6cd2388-kube-api-access-grx2k\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x\" (UID: \"92c25563-47d0-403d-afbc-0461e6cd2388\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" Nov 03 15:45:48.296986 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.296949 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c25563-47d0-403d-afbc-0461e6cd2388-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x\" (UID: \"92c25563-47d0-403d-afbc-0461e6cd2388\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" Nov 03 15:45:48.297039 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.297022 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c25563-47d0-403d-afbc-0461e6cd2388-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x\" (UID: \"92c25563-47d0-403d-afbc-0461e6cd2388\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" Nov 03 15:45:48.303892 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.303862 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:45:48.304022 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.303900 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:45:48.398872 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.398839 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c25563-47d0-403d-afbc-0461e6cd2388-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x\" (UID: \"92c25563-47d0-403d-afbc-0461e6cd2388\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" Nov 03 15:45:48.399066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.398897 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c25563-47d0-403d-afbc-0461e6cd2388-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x\" (UID: \"92c25563-47d0-403d-afbc-0461e6cd2388\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" Nov 03 15:45:48.399066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.399021 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grx2k\" (UniqueName: \"kubernetes.io/projected/92c25563-47d0-403d-afbc-0461e6cd2388-kube-api-access-grx2k\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x\" (UID: \"92c25563-47d0-403d-afbc-0461e6cd2388\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" Nov 03 15:45:48.399619 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.399587 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c25563-47d0-403d-afbc-0461e6cd2388-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x\" (UID: \"92c25563-47d0-403d-afbc-0461e6cd2388\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" Nov 03 15:45:48.401660 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.401627 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c25563-47d0-403d-afbc-0461e6cd2388-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x\" (UID: \"92c25563-47d0-403d-afbc-0461e6cd2388\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" Nov 03 15:45:48.464459 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.464384 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grx2k\" (UniqueName: \"kubernetes.io/projected/92c25563-47d0-403d-afbc-0461e6cd2388-kube-api-access-grx2k\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x\" (UID: \"92c25563-47d0-403d-afbc-0461e6cd2388\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" Nov 03 15:45:48.521077 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.521045 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" Nov 03 15:45:48.915220 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:48.915181 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x"] Nov 03 15:45:48.915838 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:45:48.915814 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c25563_47d0_403d_afbc_0461e6cd2388.slice/crio-86e9fd399202728bdc757fe9bcdbc96e909708a63f502b8ece538c63400af609 WatchSource:0}: Error finding container 86e9fd399202728bdc757fe9bcdbc96e909708a63f502b8ece538c63400af609: Status 404 returned error can't find the container with id 86e9fd399202728bdc757fe9bcdbc96e909708a63f502b8ece538c63400af609 Nov 03 15:45:49.635020 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:49.634980 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" event={"ID":"92c25563-47d0-403d-afbc-0461e6cd2388","Type":"ContainerStarted","Data":"f15fac05c3ad5562d9cb5af36e2adfe6760fbc561af7ae2613a54457a0b9c9ac"} Nov 03 15:45:49.635421 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:49.635080 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" event={"ID":"92c25563-47d0-403d-afbc-0461e6cd2388","Type":"ContainerStarted","Data":"86e9fd399202728bdc757fe9bcdbc96e909708a63f502b8ece538c63400af609"} Nov 03 15:45:49.756669 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:49.756612 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" podStartSLOduration=1.7565977 podStartE2EDuration="1.7565977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:49.752658436 +0000 UTC m=+2257.183616243" watchObservedRunningTime="2026-11-03 15:45:49.7565977 +0000 UTC m=+2257.187555506" Nov 03 15:45:50.646817 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:50.646785 4290 generic.go:356] "Generic (PLEG): container finished" podID="92c25563-47d0-403d-afbc-0461e6cd2388" containerID="f15fac05c3ad5562d9cb5af36e2adfe6760fbc561af7ae2613a54457a0b9c9ac" exitCode=0 Nov 03 15:45:50.647328 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:50.646883 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" event={"ID":"92c25563-47d0-403d-afbc-0461e6cd2388","Type":"ContainerDied","Data":"f15fac05c3ad5562d9cb5af36e2adfe6760fbc561af7ae2613a54457a0b9c9ac"} Nov 03 15:45:51.817292 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:51.817266 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" Nov 03 15:45:51.949430 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:51.949352 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c25563-47d0-403d-afbc-0461e6cd2388-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c25563-47d0-403d-afbc-0461e6cd2388-maas-api-openshift-ca\") pod \"92c25563-47d0-403d-afbc-0461e6cd2388\" (UID: \"92c25563-47d0-403d-afbc-0461e6cd2388\") " Nov 03 15:45:51.949430 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:51.949409 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c25563-47d0-403d-afbc-0461e6cd2388-kube-api-access-grx2k\" (UniqueName: \"kubernetes.io/projected/92c25563-47d0-403d-afbc-0461e6cd2388-kube-api-access-grx2k\") pod \"92c25563-47d0-403d-afbc-0461e6cd2388\" (UID: \"92c25563-47d0-403d-afbc-0461e6cd2388\") " Nov 03 15:45:51.949662 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:51.949450 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92c25563-47d0-403d-afbc-0461e6cd2388-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c25563-47d0-403d-afbc-0461e6cd2388-maas-api-xks-ca\") pod \"92c25563-47d0-403d-afbc-0461e6cd2388\" (UID: \"92c25563-47d0-403d-afbc-0461e6cd2388\") " Nov 03 15:45:51.949816 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:51.949785 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c25563-47d0-403d-afbc-0461e6cd2388-maas-api-openshift-ca" pod "92c25563-47d0-403d-afbc-0461e6cd2388" (UID: "92c25563-47d0-403d-afbc-0461e6cd2388"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:51.950462 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:51.950434 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c25563-47d0-403d-afbc-0461e6cd2388-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:51.952359 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:51.952329 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92c25563-47d0-403d-afbc-0461e6cd2388-maas-api-xks-ca" pod "92c25563-47d0-403d-afbc-0461e6cd2388" (UID: "92c25563-47d0-403d-afbc-0461e6cd2388"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:45:51.952682 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:51.952657 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c25563-47d0-403d-afbc-0461e6cd2388-kube-api-access-grx2k" pod "92c25563-47d0-403d-afbc-0461e6cd2388" (UID: "92c25563-47d0-403d-afbc-0461e6cd2388"). InnerVolumeSpecName "kube-api-access-grx2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:52.051777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:52.051741 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grx2k\" (UniqueName: \"kubernetes.io/projected/92c25563-47d0-403d-afbc-0461e6cd2388-kube-api-access-grx2k\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:52.051777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:52.051772 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c25563-47d0-403d-afbc-0461e6cd2388-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:52.667786 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:52.667757 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" Nov 03 15:45:52.667786 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:52.667779 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x" event={"ID":"92c25563-47d0-403d-afbc-0461e6cd2388","Type":"ContainerDied","Data":"86e9fd399202728bdc757fe9bcdbc96e909708a63f502b8ece538c63400af609"} Nov 03 15:45:52.668101 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:52.667840 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86e9fd399202728bdc757fe9bcdbc96e909708a63f502b8ece538c63400af609" Nov 03 15:45:54.299453 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.299416 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv"] Nov 03 15:45:54.299918 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.299651 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" podUID="69325d2c-8a89-47f0-be5f-f68ff639d398" containerName="maas-api" containerID="cri-o://fc4e67b6ada6953150351a01a0decd4faf9fb57b5606d53d8055923abb4e268c" gracePeriod=30 Nov 03 15:45:54.613216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.613185 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:45:54.689919 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.689884 4290 generic.go:356] "Generic (PLEG): container finished" podID="69325d2c-8a89-47f0-be5f-f68ff639d398" containerID="fc4e67b6ada6953150351a01a0decd4faf9fb57b5606d53d8055923abb4e268c" exitCode=0 Nov 03 15:45:54.690102 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.689932 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" Nov 03 15:45:54.690102 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.689980 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" event={"ID":"69325d2c-8a89-47f0-be5f-f68ff639d398","Type":"ContainerDied","Data":"fc4e67b6ada6953150351a01a0decd4faf9fb57b5606d53d8055923abb4e268c"} Nov 03 15:45:54.690102 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.690054 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv" event={"ID":"69325d2c-8a89-47f0-be5f-f68ff639d398","Type":"ContainerDied","Data":"9960a24ddc301103b3eb574b59245e23131713f68dec891d8ca0a408251580ef"} Nov 03 15:45:54.690102 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.690088 4290 scope.go:122] "RemoveContainer" containerID="fc4e67b6ada6953150351a01a0decd4faf9fb57b5606d53d8055923abb4e268c" Nov 03 15:45:54.703071 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.703045 4290 scope.go:122] "RemoveContainer" containerID="fc4e67b6ada6953150351a01a0decd4faf9fb57b5606d53d8055923abb4e268c" Nov 03 15:45:54.704776 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:45:54.704743 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4e67b6ada6953150351a01a0decd4faf9fb57b5606d53d8055923abb4e268c\": container with ID starting with fc4e67b6ada6953150351a01a0decd4faf9fb57b5606d53d8055923abb4e268c not found: ID does not exist" containerID="fc4e67b6ada6953150351a01a0decd4faf9fb57b5606d53d8055923abb4e268c" Nov 03 15:45:54.704878 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.704787 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4e67b6ada6953150351a01a0decd4faf9fb57b5606d53d8055923abb4e268c"} err="failed to get container status \"fc4e67b6ada6953150351a01a0decd4faf9fb57b5606d53d8055923abb4e268c\": rpc error: code = NotFound desc = could not find container \"fc4e67b6ada6953150351a01a0decd4faf9fb57b5606d53d8055923abb4e268c\": container with ID starting with fc4e67b6ada6953150351a01a0decd4faf9fb57b5606d53d8055923abb4e268c not found: ID does not exist" Nov 03 15:45:54.796329 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.796295 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69325d2c-8a89-47f0-be5f-f68ff639d398-kube-api-access-86fhb\" (UniqueName: \"kubernetes.io/projected/69325d2c-8a89-47f0-be5f-f68ff639d398-kube-api-access-86fhb\") pod \"69325d2c-8a89-47f0-be5f-f68ff639d398\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " Nov 03 15:45:54.796521 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.796388 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69325d2c-8a89-47f0-be5f-f68ff639d398-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69325d2c-8a89-47f0-be5f-f68ff639d398-openshift-service-ca\") pod \"69325d2c-8a89-47f0-be5f-f68ff639d398\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " Nov 03 15:45:54.796521 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.796435 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-metrics-certs\") pod \"69325d2c-8a89-47f0-be5f-f68ff639d398\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " Nov 03 15:45:54.796521 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.796471 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-maas-api-tls\") pod \"69325d2c-8a89-47f0-be5f-f68ff639d398\" (UID: \"69325d2c-8a89-47f0-be5f-f68ff639d398\") " Nov 03 15:45:54.797501 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.797465 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69325d2c-8a89-47f0-be5f-f68ff639d398-openshift-service-ca" pod "69325d2c-8a89-47f0-be5f-f68ff639d398" (UID: "69325d2c-8a89-47f0-be5f-f68ff639d398"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:54.801209 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.801167 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69325d2c-8a89-47f0-be5f-f68ff639d398-kube-api-access-86fhb" pod "69325d2c-8a89-47f0-be5f-f68ff639d398" (UID: "69325d2c-8a89-47f0-be5f-f68ff639d398"). InnerVolumeSpecName "kube-api-access-86fhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:54.801383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.801236 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-metrics-certs" pod "69325d2c-8a89-47f0-be5f-f68ff639d398" (UID: "69325d2c-8a89-47f0-be5f-f68ff639d398"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:45:54.832060 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.819839 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-maas-api-tls" pod "69325d2c-8a89-47f0-be5f-f68ff639d398" (UID: "69325d2c-8a89-47f0-be5f-f68ff639d398"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:45:54.912725 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.904042 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86fhb\" (UniqueName: \"kubernetes.io/projected/69325d2c-8a89-47f0-be5f-f68ff639d398-kube-api-access-86fhb\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:54.912725 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.904091 4290 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69325d2c-8a89-47f0-be5f-f68ff639d398-openshift-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:54.912725 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.904110 4290 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-metrics-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:54.912725 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:54.904124 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69325d2c-8a89-47f0-be5f-f68ff639d398-maas-api-tls\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:55.117767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:55.117665 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv"] Nov 03 15:45:55.165129 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:55.165089 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-7xpgv"] Nov 03 15:45:56.579562 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:56.579520 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj"] Nov 03 15:45:56.626910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:56.626878 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cad072f-7scdj"] Nov 03 15:45:56.682335 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:56.682292 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="218b350b-786d-4f94-a3a4-dbfb11c77e0e" path="/var/lib/kubelet/pods/218b350b-786d-4f94-a3a4-dbfb11c77e0e/volumes" Nov 03 15:45:56.682983 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:56.682962 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69325d2c-8a89-47f0-be5f-f68ff639d398" path="/var/lib/kubelet/pods/69325d2c-8a89-47f0-be5f-f68ff639d398/volumes" Nov 03 15:45:58.108267 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:58.108230 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d"] Nov 03 15:45:58.222031 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:45:58.221981 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc"] Nov 03 15:46:00.733197 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:00.733163 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r"] Nov 03 15:46:03.109047 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.108996 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" podUID="65b4e93f-0dce-4a32-9248-35979203ae23" containerName="payload-processing" containerID="cri-o://81bce026ba38af283bc21060ea75b21f7a3986e4915bbc7a4c02790cf2bbff8b" gracePeriod=25 Nov 03 15:46:03.222598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.222537 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" podUID="bd9a4e78-c549-4d50-8b0a-05552100f91f" containerName="payload-pre-processing" containerID="cri-o://cfca423a50b3380e76176299a611ea82331250ad443b8bed91b5a780cb263f38" gracePeriod=25 Nov 03 15:46:03.433765 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.433739 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" Nov 03 15:46:03.488670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.488621 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b4e93f-0dce-4a32-9248-35979203ae23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b4e93f-0dce-4a32-9248-35979203ae23-plugins-config-volume\") pod \"65b4e93f-0dce-4a32-9248-35979203ae23\" (UID: \"65b4e93f-0dce-4a32-9248-35979203ae23\") " Nov 03 15:46:03.488890 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.488815 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b4e93f-0dce-4a32-9248-35979203ae23-kube-api-access-d5vjb\" (UniqueName: \"kubernetes.io/projected/65b4e93f-0dce-4a32-9248-35979203ae23-kube-api-access-d5vjb\") pod \"65b4e93f-0dce-4a32-9248-35979203ae23\" (UID: \"65b4e93f-0dce-4a32-9248-35979203ae23\") " Nov 03 15:46:03.489337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.489052 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b4e93f-0dce-4a32-9248-35979203ae23-plugins-config-volume" pod "65b4e93f-0dce-4a32-9248-35979203ae23" (UID: "65b4e93f-0dce-4a32-9248-35979203ae23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:03.489337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.489309 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b4e93f-0dce-4a32-9248-35979203ae23-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:03.491679 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.491653 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b4e93f-0dce-4a32-9248-35979203ae23-kube-api-access-d5vjb" pod "65b4e93f-0dce-4a32-9248-35979203ae23" (UID: "65b4e93f-0dce-4a32-9248-35979203ae23"). InnerVolumeSpecName "kube-api-access-d5vjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:03.533903 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.533878 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" Nov 03 15:46:03.590571 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.590531 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd9a4e78-c549-4d50-8b0a-05552100f91f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd9a4e78-c549-4d50-8b0a-05552100f91f-plugins-config-volume\") pod \"bd9a4e78-c549-4d50-8b0a-05552100f91f\" (UID: \"bd9a4e78-c549-4d50-8b0a-05552100f91f\") " Nov 03 15:46:03.590836 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.590719 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9a4e78-c549-4d50-8b0a-05552100f91f-kube-api-access-9vd4p\" (UniqueName: \"kubernetes.io/projected/bd9a4e78-c549-4d50-8b0a-05552100f91f-kube-api-access-9vd4p\") pod \"bd9a4e78-c549-4d50-8b0a-05552100f91f\" (UID: \"bd9a4e78-c549-4d50-8b0a-05552100f91f\") " Nov 03 15:46:03.591231 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.591204 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9a4e78-c549-4d50-8b0a-05552100f91f-plugins-config-volume" pod "bd9a4e78-c549-4d50-8b0a-05552100f91f" (UID: "bd9a4e78-c549-4d50-8b0a-05552100f91f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:03.591475 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.591450 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd9a4e78-c549-4d50-8b0a-05552100f91f-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:03.591475 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.591474 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5vjb\" (UniqueName: \"kubernetes.io/projected/65b4e93f-0dce-4a32-9248-35979203ae23-kube-api-access-d5vjb\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:03.593038 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.593015 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9a4e78-c549-4d50-8b0a-05552100f91f-kube-api-access-9vd4p" pod "bd9a4e78-c549-4d50-8b0a-05552100f91f" (UID: "bd9a4e78-c549-4d50-8b0a-05552100f91f"). InnerVolumeSpecName "kube-api-access-9vd4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:03.692964 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.692878 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vd4p\" (UniqueName: \"kubernetes.io/projected/bd9a4e78-c549-4d50-8b0a-05552100f91f-kube-api-access-9vd4p\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:03.788924 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.788888 4290 generic.go:356] "Generic (PLEG): container finished" podID="bd9a4e78-c549-4d50-8b0a-05552100f91f" containerID="cfca423a50b3380e76176299a611ea82331250ad443b8bed91b5a780cb263f38" exitCode=0 Nov 03 15:46:03.789117 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.788940 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" Nov 03 15:46:03.789117 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.788974 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" event={"ID":"bd9a4e78-c549-4d50-8b0a-05552100f91f","Type":"ContainerDied","Data":"cfca423a50b3380e76176299a611ea82331250ad443b8bed91b5a780cb263f38"} Nov 03 15:46:03.789117 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.789029 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc" event={"ID":"bd9a4e78-c549-4d50-8b0a-05552100f91f","Type":"ContainerDied","Data":"684e43adde07e6072086c76cc35a3e63fc27b5a1bfbb520eb3656098f17f1a7e"} Nov 03 15:46:03.789117 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.789062 4290 scope.go:122] "RemoveContainer" containerID="cfca423a50b3380e76176299a611ea82331250ad443b8bed91b5a780cb263f38" Nov 03 15:46:03.791076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.791051 4290 generic.go:356] "Generic (PLEG): container finished" podID="65b4e93f-0dce-4a32-9248-35979203ae23" containerID="81bce026ba38af283bc21060ea75b21f7a3986e4915bbc7a4c02790cf2bbff8b" exitCode=0 Nov 03 15:46:03.791208 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.791104 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" Nov 03 15:46:03.791208 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.791148 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" event={"ID":"65b4e93f-0dce-4a32-9248-35979203ae23","Type":"ContainerDied","Data":"81bce026ba38af283bc21060ea75b21f7a3986e4915bbc7a4c02790cf2bbff8b"} Nov 03 15:46:03.791312 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.791224 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d" event={"ID":"65b4e93f-0dce-4a32-9248-35979203ae23","Type":"ContainerDied","Data":"523489e0a87d7f68b5dc4c59ee75f0b882538778911dd1e17764aa1ba60caf2c"} Nov 03 15:46:03.801417 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.801399 4290 scope.go:122] "RemoveContainer" containerID="cfca423a50b3380e76176299a611ea82331250ad443b8bed91b5a780cb263f38" Nov 03 15:46:03.801842 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:46:03.801734 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfca423a50b3380e76176299a611ea82331250ad443b8bed91b5a780cb263f38\": container with ID starting with cfca423a50b3380e76176299a611ea82331250ad443b8bed91b5a780cb263f38 not found: ID does not exist" containerID="cfca423a50b3380e76176299a611ea82331250ad443b8bed91b5a780cb263f38" Nov 03 15:46:03.801842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.801762 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfca423a50b3380e76176299a611ea82331250ad443b8bed91b5a780cb263f38"} err="failed to get container status \"cfca423a50b3380e76176299a611ea82331250ad443b8bed91b5a780cb263f38\": rpc error: code = NotFound desc = could not find container \"cfca423a50b3380e76176299a611ea82331250ad443b8bed91b5a780cb263f38\": container with ID starting with cfca423a50b3380e76176299a611ea82331250ad443b8bed91b5a780cb263f38 not found: ID does not exist" Nov 03 15:46:03.801842 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.801781 4290 scope.go:122] "RemoveContainer" containerID="81bce026ba38af283bc21060ea75b21f7a3986e4915bbc7a4c02790cf2bbff8b" Nov 03 15:46:03.811902 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.811885 4290 scope.go:122] "RemoveContainer" containerID="81bce026ba38af283bc21060ea75b21f7a3986e4915bbc7a4c02790cf2bbff8b" Nov 03 15:46:03.812177 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:46:03.812157 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81bce026ba38af283bc21060ea75b21f7a3986e4915bbc7a4c02790cf2bbff8b\": container with ID starting with 81bce026ba38af283bc21060ea75b21f7a3986e4915bbc7a4c02790cf2bbff8b not found: ID does not exist" containerID="81bce026ba38af283bc21060ea75b21f7a3986e4915bbc7a4c02790cf2bbff8b" Nov 03 15:46:03.812226 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.812186 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81bce026ba38af283bc21060ea75b21f7a3986e4915bbc7a4c02790cf2bbff8b"} err="failed to get container status \"81bce026ba38af283bc21060ea75b21f7a3986e4915bbc7a4c02790cf2bbff8b\": rpc error: code = NotFound desc = could not find container \"81bce026ba38af283bc21060ea75b21f7a3986e4915bbc7a4c02790cf2bbff8b\": container with ID starting with 81bce026ba38af283bc21060ea75b21f7a3986e4915bbc7a4c02790cf2bbff8b not found: ID does not exist" Nov 03 15:46:03.965747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.965632 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc"] Nov 03 15:46:03.972332 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:03.972304 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9855ff-7764c48455-sjxfc"] Nov 03 15:46:04.074661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:04.074549 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d"] Nov 03 15:46:04.079206 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:04.079167 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9855ff-7859668d95-shd5d"] Nov 03 15:46:04.681884 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:04.681850 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b4e93f-0dce-4a32-9248-35979203ae23" path="/var/lib/kubelet/pods/65b4e93f-0dce-4a32-9248-35979203ae23/volumes" Nov 03 15:46:04.682471 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:04.682451 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9a4e78-c549-4d50-8b0a-05552100f91f" path="/var/lib/kubelet/pods/bd9a4e78-c549-4d50-8b0a-05552100f91f/volumes" Nov 03 15:46:05.733639 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:05.733584 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" podUID="0041b317-bb1c-4a54-ae90-ac6078087205" containerName="payload-pre-processing" containerID="cri-o://681778167ad5fdc1d480ed3feeddc655672567fe1dbc7617c598bd293cdea5f2" gracePeriod=25 Nov 03 15:46:06.025082 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:06.025051 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" Nov 03 15:46:06.126133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:06.126099 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0041b317-bb1c-4a54-ae90-ac6078087205-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0041b317-bb1c-4a54-ae90-ac6078087205-plugins-config-volume\") pod \"0041b317-bb1c-4a54-ae90-ac6078087205\" (UID: \"0041b317-bb1c-4a54-ae90-ac6078087205\") " Nov 03 15:46:06.126519 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:06.126478 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0041b317-bb1c-4a54-ae90-ac6078087205-kube-api-access-fvdn2\" (UniqueName: \"kubernetes.io/projected/0041b317-bb1c-4a54-ae90-ac6078087205-kube-api-access-fvdn2\") pod \"0041b317-bb1c-4a54-ae90-ac6078087205\" (UID: \"0041b317-bb1c-4a54-ae90-ac6078087205\") " Nov 03 15:46:06.126781 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:06.126736 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0041b317-bb1c-4a54-ae90-ac6078087205-plugins-config-volume" pod "0041b317-bb1c-4a54-ae90-ac6078087205" (UID: "0041b317-bb1c-4a54-ae90-ac6078087205"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:06.141769 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:06.127380 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0041b317-bb1c-4a54-ae90-ac6078087205-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:06.148726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:06.148514 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0041b317-bb1c-4a54-ae90-ac6078087205-kube-api-access-fvdn2" pod "0041b317-bb1c-4a54-ae90-ac6078087205" (UID: "0041b317-bb1c-4a54-ae90-ac6078087205"). InnerVolumeSpecName "kube-api-access-fvdn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:06.229295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:06.229264 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvdn2\" (UniqueName: \"kubernetes.io/projected/0041b317-bb1c-4a54-ae90-ac6078087205-kube-api-access-fvdn2\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:06.819187 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:06.819153 4290 generic.go:356] "Generic (PLEG): container finished" podID="0041b317-bb1c-4a54-ae90-ac6078087205" containerID="681778167ad5fdc1d480ed3feeddc655672567fe1dbc7617c598bd293cdea5f2" exitCode=0 Nov 03 15:46:06.819621 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:06.819211 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" Nov 03 15:46:06.819621 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:06.819241 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" event={"ID":"0041b317-bb1c-4a54-ae90-ac6078087205","Type":"ContainerDied","Data":"681778167ad5fdc1d480ed3feeddc655672567fe1dbc7617c598bd293cdea5f2"} Nov 03 15:46:06.819621 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:06.819297 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r" event={"ID":"0041b317-bb1c-4a54-ae90-ac6078087205","Type":"ContainerDied","Data":"1bf381844f1a121bcfc7c075637e784ea6545698133efca661d13d80d02dce27"} Nov 03 15:46:06.819621 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:06.819319 4290 scope.go:122] "RemoveContainer" containerID="681778167ad5fdc1d480ed3feeddc655672567fe1dbc7617c598bd293cdea5f2" Nov 03 15:46:06.830593 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:06.830570 4290 scope.go:122] "RemoveContainer" containerID="681778167ad5fdc1d480ed3feeddc655672567fe1dbc7617c598bd293cdea5f2" Nov 03 15:46:06.830891 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:46:06.830868 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"681778167ad5fdc1d480ed3feeddc655672567fe1dbc7617c598bd293cdea5f2\": container with ID starting with 681778167ad5fdc1d480ed3feeddc655672567fe1dbc7617c598bd293cdea5f2 not found: ID does not exist" containerID="681778167ad5fdc1d480ed3feeddc655672567fe1dbc7617c598bd293cdea5f2" Nov 03 15:46:06.830950 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:06.830899 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"681778167ad5fdc1d480ed3feeddc655672567fe1dbc7617c598bd293cdea5f2"} err="failed to get container status \"681778167ad5fdc1d480ed3feeddc655672567fe1dbc7617c598bd293cdea5f2\": rpc error: code = NotFound desc = could not find container \"681778167ad5fdc1d480ed3feeddc655672567fe1dbc7617c598bd293cdea5f2\": container with ID starting with 681778167ad5fdc1d480ed3feeddc655672567fe1dbc7617c598bd293cdea5f2 not found: ID does not exist" Nov 03 15:46:06.991150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:06.991056 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r"] Nov 03 15:46:07.038092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:07.038063 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-78034f-847f8cb9f5-gm74r"] Nov 03 15:46:07.055287 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:07.055251 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g"] Nov 03 15:46:07.059516 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:07.059492 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0dd9a75-8hc4g"] Nov 03 15:46:08.682410 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:08.682331 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0041b317-bb1c-4a54-ae90-ac6078087205" path="/var/lib/kubelet/pods/0041b317-bb1c-4a54-ae90-ac6078087205/volumes" Nov 03 15:46:08.682991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:08.682975 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61fa98e-f543-4c48-91ad-9abccd424a9d" path="/var/lib/kubelet/pods/d61fa98e-f543-4c48-91ad-9abccd424a9d/volumes" Nov 03 15:46:08.706933 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:46:08.706896 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:46:17.382521 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:17.382485 4290 scope.go:122] "RemoveContainer" containerID="c03b42a2a365210985cbdd4fb254c6c91119e73a9ca8fe9a03ef4c47817e84b4" Nov 03 15:46:17.409210 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:17.409186 4290 scope.go:122] "RemoveContainer" containerID="58bde94461003086ea5ac212a6c1c3c098599adb37cfe245923b666d5f51b70e" Nov 03 15:46:21.171362 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:21.171111 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl"] Nov 03 15:46:21.216933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:21.216898 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9"] Nov 03 15:46:23.111737 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:23.111570 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42"] Nov 03 15:46:23.161405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:23.161362 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c79915bb-9rz42"] Nov 03 15:46:24.771059 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:24.771019 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d04a5ca-1479-4cb9-992d-a296d7ecbd2d" path="/var/lib/kubelet/pods/3d04a5ca-1479-4cb9-992d-a296d7ecbd2d/volumes" Nov 03 15:46:25.943729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:25.943681 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x"] Nov 03 15:46:25.988670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:25.988637 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-78034f-q8p7x"] Nov 03 15:46:26.681559 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:26.681528 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c25563-47d0-403d-afbc-0461e6cd2388" path="/var/lib/kubelet/pods/92c25563-47d0-403d-afbc-0461e6cd2388/volumes" Nov 03 15:46:28.426753 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:28.426716 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn"] Nov 03 15:46:28.538295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:28.538257 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw"] Nov 03 15:46:32.236347 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.236302 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql"] Nov 03 15:46:32.237090 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237072 4290 cpu_manager.go:414] "containerMap: removing container" podUID="65b4e93f-0dce-4a32-9248-35979203ae23" containerName="payload-processing" Nov 03 15:46:32.237150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237092 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b4e93f-0dce-4a32-9248-35979203ae23" containerName="payload-processing" Nov 03 15:46:32.237150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237112 4290 cpu_manager.go:414] "containerMap: removing container" podUID="0041b317-bb1c-4a54-ae90-ac6078087205" containerName="payload-pre-processing" Nov 03 15:46:32.237150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237121 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0041b317-bb1c-4a54-ae90-ac6078087205" containerName="payload-pre-processing" Nov 03 15:46:32.237150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237131 4290 cpu_manager.go:414] "containerMap: removing container" podUID="bd9a4e78-c549-4d50-8b0a-05552100f91f" containerName="payload-pre-processing" Nov 03 15:46:32.237150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237136 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9a4e78-c549-4d50-8b0a-05552100f91f" containerName="payload-pre-processing" Nov 03 15:46:32.237150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237146 4290 cpu_manager.go:414] "containerMap: removing container" podUID="8548a302-2c94-4f27-89c0-653e460d571a" containerName="cleanup" Nov 03 15:46:32.237150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237150 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8548a302-2c94-4f27-89c0-653e460d571a" containerName="cleanup" Nov 03 15:46:32.237383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237162 4290 cpu_manager.go:414] "containerMap: removing container" podUID="be07c854-6c6c-4760-a22e-7b28af22ae21" containerName="cleanup" Nov 03 15:46:32.237383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237166 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be07c854-6c6c-4760-a22e-7b28af22ae21" containerName="cleanup" Nov 03 15:46:32.237383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237177 4290 cpu_manager.go:414] "containerMap: removing container" podUID="92c25563-47d0-403d-afbc-0461e6cd2388" containerName="revoke-keys" Nov 03 15:46:32.237383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237182 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c25563-47d0-403d-afbc-0461e6cd2388" containerName="revoke-keys" Nov 03 15:46:32.237383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237207 4290 cpu_manager.go:414] "containerMap: removing container" podUID="69325d2c-8a89-47f0-be5f-f68ff639d398" containerName="maas-api" Nov 03 15:46:32.237383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237212 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69325d2c-8a89-47f0-be5f-f68ff639d398" containerName="maas-api" Nov 03 15:46:32.237383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237315 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="321f6b11-c456-4a61-9649-b3a37398086f" containerName="cleanup" Nov 03 15:46:32.237383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237331 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c25563-47d0-403d-afbc-0461e6cd2388" containerName="revoke-keys" Nov 03 15:46:32.237383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237341 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="69325d2c-8a89-47f0-be5f-f68ff639d398" containerName="maas-api" Nov 03 15:46:32.237383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237356 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b4e93f-0dce-4a32-9248-35979203ae23" containerName="payload-processing" Nov 03 15:46:32.237383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237364 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="8548a302-2c94-4f27-89c0-653e460d571a" containerName="cleanup" Nov 03 15:46:32.237383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237369 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="0041b317-bb1c-4a54-ae90-ac6078087205" containerName="payload-pre-processing" Nov 03 15:46:32.237383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237374 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9a4e78-c549-4d50-8b0a-05552100f91f" containerName="payload-pre-processing" Nov 03 15:46:32.237383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.237381 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2094a7-5e86-43c9-8e0c-917426278359" containerName="cleanup" Nov 03 15:46:32.243075 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.243056 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" Nov 03 15:46:32.341849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.341814 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql"] Nov 03 15:46:32.355632 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.355591 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:46:32.355822 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.355688 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:46:32.364543 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.364495 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d86146d-znnql\" (UID: \"d67ad11f-ae4c-49ca-9763-d2eef45ac27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" Nov 03 15:46:32.364750 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.364720 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d86146d-znnql\" (UID: \"d67ad11f-ae4c-49ca-9763-d2eef45ac27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" Nov 03 15:46:32.364991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.364967 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmrnf\" (UniqueName: \"kubernetes.io/projected/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-kube-api-access-gmrnf\") pod \"maas-api-revoke-sub-8d86146d-znnql\" (UID: \"d67ad11f-ae4c-49ca-9763-d2eef45ac27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" Nov 03 15:46:32.466142 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.466107 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmrnf\" (UniqueName: \"kubernetes.io/projected/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-kube-api-access-gmrnf\") pod \"maas-api-revoke-sub-8d86146d-znnql\" (UID: \"d67ad11f-ae4c-49ca-9763-d2eef45ac27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" Nov 03 15:46:32.466347 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.466171 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d86146d-znnql\" (UID: \"d67ad11f-ae4c-49ca-9763-d2eef45ac27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" Nov 03 15:46:32.466347 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.466283 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d86146d-znnql\" (UID: \"d67ad11f-ae4c-49ca-9763-d2eef45ac27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" Nov 03 15:46:32.467086 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.467057 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d86146d-znnql\" (UID: \"d67ad11f-ae4c-49ca-9763-d2eef45ac27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" Nov 03 15:46:32.468972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.468916 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d86146d-znnql\" (UID: \"d67ad11f-ae4c-49ca-9763-d2eef45ac27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" Nov 03 15:46:32.547033 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.546954 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmrnf\" (UniqueName: \"kubernetes.io/projected/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-kube-api-access-gmrnf\") pod \"maas-api-revoke-sub-8d86146d-znnql\" (UID: \"d67ad11f-ae4c-49ca-9763-d2eef45ac27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" Nov 03 15:46:32.560287 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:32.560251 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" Nov 03 15:46:33.005562 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.005523 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql"] Nov 03 15:46:33.077113 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:46:33.077084 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67ad11f_ae4c_49ca_9763_d2eef45ac27d.slice/crio-4b92a02684b33893a16a0745d3b592585b928fd0aeb9ae26ea687e29ef8b1e8c WatchSource:0}: Error finding container 4b92a02684b33893a16a0745d3b592585b928fd0aeb9ae26ea687e29ef8b1e8c: Status 404 returned error can't find the container with id 4b92a02684b33893a16a0745d3b592585b928fd0aeb9ae26ea687e29ef8b1e8c Nov 03 15:46:33.428189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.427237 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" podUID="b8671d1f-ea5b-4433-af1d-40b53480728b" containerName="payload-processing" containerID="cri-o://a975247eb2830e50582e3a1e29cd375853ee5c2c5782436473ee8d6f2790d0d4" gracePeriod=25 Nov 03 15:46:33.539321 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.539257 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" podUID="7c74b1f4-9c5f-4909-bb4e-92181b087fb2" containerName="payload-pre-processing" containerID="cri-o://fbef979bc24746a8e15b5f8fc073c1f5ddd280b1e9452b4ccf951736467eda67" gracePeriod=25 Nov 03 15:46:33.759994 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.759964 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" Nov 03 15:46:33.792505 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.792476 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8671d1f-ea5b-4433-af1d-40b53480728b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8671d1f-ea5b-4433-af1d-40b53480728b-plugins-config-volume\") pod \"b8671d1f-ea5b-4433-af1d-40b53480728b\" (UID: \"b8671d1f-ea5b-4433-af1d-40b53480728b\") " Nov 03 15:46:33.792505 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.792510 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8671d1f-ea5b-4433-af1d-40b53480728b-kube-api-access-k2nzl\" (UniqueName: \"kubernetes.io/projected/b8671d1f-ea5b-4433-af1d-40b53480728b-kube-api-access-k2nzl\") pod \"b8671d1f-ea5b-4433-af1d-40b53480728b\" (UID: \"b8671d1f-ea5b-4433-af1d-40b53480728b\") " Nov 03 15:46:33.792994 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.792960 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8671d1f-ea5b-4433-af1d-40b53480728b-plugins-config-volume" pod "b8671d1f-ea5b-4433-af1d-40b53480728b" (UID: "b8671d1f-ea5b-4433-af1d-40b53480728b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:33.795083 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.795015 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8671d1f-ea5b-4433-af1d-40b53480728b-kube-api-access-k2nzl" pod "b8671d1f-ea5b-4433-af1d-40b53480728b" (UID: "b8671d1f-ea5b-4433-af1d-40b53480728b"). InnerVolumeSpecName "kube-api-access-k2nzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:33.860566 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.860535 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" Nov 03 15:46:33.893850 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.893814 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c74b1f4-9c5f-4909-bb4e-92181b087fb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c74b1f4-9c5f-4909-bb4e-92181b087fb2-plugins-config-volume\") pod \"7c74b1f4-9c5f-4909-bb4e-92181b087fb2\" (UID: \"7c74b1f4-9c5f-4909-bb4e-92181b087fb2\") " Nov 03 15:46:33.894126 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.894057 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c74b1f4-9c5f-4909-bb4e-92181b087fb2-kube-api-access-mpf25\" (UniqueName: \"kubernetes.io/projected/7c74b1f4-9c5f-4909-bb4e-92181b087fb2-kube-api-access-mpf25\") pod \"7c74b1f4-9c5f-4909-bb4e-92181b087fb2\" (UID: \"7c74b1f4-9c5f-4909-bb4e-92181b087fb2\") " Nov 03 15:46:33.894275 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.894250 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c74b1f4-9c5f-4909-bb4e-92181b087fb2-plugins-config-volume" pod "7c74b1f4-9c5f-4909-bb4e-92181b087fb2" (UID: "7c74b1f4-9c5f-4909-bb4e-92181b087fb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:33.894584 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.894567 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c74b1f4-9c5f-4909-bb4e-92181b087fb2-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:33.894690 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.894590 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8671d1f-ea5b-4433-af1d-40b53480728b-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:33.894690 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.894604 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2nzl\" (UniqueName: \"kubernetes.io/projected/b8671d1f-ea5b-4433-af1d-40b53480728b-kube-api-access-k2nzl\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:33.896443 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.896416 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c74b1f4-9c5f-4909-bb4e-92181b087fb2-kube-api-access-mpf25" pod "7c74b1f4-9c5f-4909-bb4e-92181b087fb2" (UID: "7c74b1f4-9c5f-4909-bb4e-92181b087fb2"). InnerVolumeSpecName "kube-api-access-mpf25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:33.995759 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:33.995681 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpf25\" (UniqueName: \"kubernetes.io/projected/7c74b1f4-9c5f-4909-bb4e-92181b087fb2-kube-api-access-mpf25\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:34.066084 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.066052 4290 generic.go:356] "Generic (PLEG): container finished" podID="b8671d1f-ea5b-4433-af1d-40b53480728b" containerID="a975247eb2830e50582e3a1e29cd375853ee5c2c5782436473ee8d6f2790d0d4" exitCode=0 Nov 03 15:46:34.066298 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.066153 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" Nov 03 15:46:34.066298 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.066195 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" event={"ID":"b8671d1f-ea5b-4433-af1d-40b53480728b","Type":"ContainerDied","Data":"a975247eb2830e50582e3a1e29cd375853ee5c2c5782436473ee8d6f2790d0d4"} Nov 03 15:46:34.066298 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.066256 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn" event={"ID":"b8671d1f-ea5b-4433-af1d-40b53480728b","Type":"ContainerDied","Data":"572f285ba07f23b95305bb69c12874ad11fb2988df2e51fb0a2faf75fbe3b6b4"} Nov 03 15:46:34.066298 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.066287 4290 scope.go:122] "RemoveContainer" containerID="a975247eb2830e50582e3a1e29cd375853ee5c2c5782436473ee8d6f2790d0d4" Nov 03 15:46:34.072081 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.071014 4290 generic.go:356] "Generic (PLEG): container finished" podID="7c74b1f4-9c5f-4909-bb4e-92181b087fb2" containerID="fbef979bc24746a8e15b5f8fc073c1f5ddd280b1e9452b4ccf951736467eda67" exitCode=0 Nov 03 15:46:34.072081 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.071069 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" Nov 03 15:46:34.072081 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.071103 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" event={"ID":"7c74b1f4-9c5f-4909-bb4e-92181b087fb2","Type":"ContainerDied","Data":"fbef979bc24746a8e15b5f8fc073c1f5ddd280b1e9452b4ccf951736467eda67"} Nov 03 15:46:34.072081 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.071162 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw" event={"ID":"7c74b1f4-9c5f-4909-bb4e-92181b087fb2","Type":"ContainerDied","Data":"b49446805d83fed235db9c709f874ad5264968f35ec65813b6615bff237e77b9"} Nov 03 15:46:34.081300 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.080123 4290 generic.go:356] "Generic (PLEG): container finished" podID="d67ad11f-ae4c-49ca-9763-d2eef45ac27d" containerID="9b5bb1f39a0497e31390e3de4e8850592c0f2bbbf0ffd121e6b913a33e1cfea2" exitCode=0 Nov 03 15:46:34.081300 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.080180 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" event={"ID":"d67ad11f-ae4c-49ca-9763-d2eef45ac27d","Type":"ContainerDied","Data":"9b5bb1f39a0497e31390e3de4e8850592c0f2bbbf0ffd121e6b913a33e1cfea2"} Nov 03 15:46:34.081300 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.080224 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" event={"ID":"d67ad11f-ae4c-49ca-9763-d2eef45ac27d","Type":"ContainerStarted","Data":"4b92a02684b33893a16a0745d3b592585b928fd0aeb9ae26ea687e29ef8b1e8c"} Nov 03 15:46:34.084747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.084693 4290 scope.go:122] "RemoveContainer" containerID="a975247eb2830e50582e3a1e29cd375853ee5c2c5782436473ee8d6f2790d0d4" Nov 03 15:46:34.085299 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:46:34.085147 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a975247eb2830e50582e3a1e29cd375853ee5c2c5782436473ee8d6f2790d0d4\": container with ID starting with a975247eb2830e50582e3a1e29cd375853ee5c2c5782436473ee8d6f2790d0d4 not found: ID does not exist" containerID="a975247eb2830e50582e3a1e29cd375853ee5c2c5782436473ee8d6f2790d0d4" Nov 03 15:46:34.085299 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.085183 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a975247eb2830e50582e3a1e29cd375853ee5c2c5782436473ee8d6f2790d0d4"} err="failed to get container status \"a975247eb2830e50582e3a1e29cd375853ee5c2c5782436473ee8d6f2790d0d4\": rpc error: code = NotFound desc = could not find container \"a975247eb2830e50582e3a1e29cd375853ee5c2c5782436473ee8d6f2790d0d4\": container with ID starting with a975247eb2830e50582e3a1e29cd375853ee5c2c5782436473ee8d6f2790d0d4 not found: ID does not exist" Nov 03 15:46:34.085299 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.085209 4290 scope.go:122] "RemoveContainer" containerID="fbef979bc24746a8e15b5f8fc073c1f5ddd280b1e9452b4ccf951736467eda67" Nov 03 15:46:34.098316 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.098296 4290 scope.go:122] "RemoveContainer" containerID="fbef979bc24746a8e15b5f8fc073c1f5ddd280b1e9452b4ccf951736467eda67" Nov 03 15:46:34.098620 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:46:34.098599 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbef979bc24746a8e15b5f8fc073c1f5ddd280b1e9452b4ccf951736467eda67\": container with ID starting with fbef979bc24746a8e15b5f8fc073c1f5ddd280b1e9452b4ccf951736467eda67 not found: ID does not exist" containerID="fbef979bc24746a8e15b5f8fc073c1f5ddd280b1e9452b4ccf951736467eda67" Nov 03 15:46:34.098668 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.098629 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbef979bc24746a8e15b5f8fc073c1f5ddd280b1e9452b4ccf951736467eda67"} err="failed to get container status \"fbef979bc24746a8e15b5f8fc073c1f5ddd280b1e9452b4ccf951736467eda67\": rpc error: code = NotFound desc = could not find container \"fbef979bc24746a8e15b5f8fc073c1f5ddd280b1e9452b4ccf951736467eda67\": container with ID starting with fbef979bc24746a8e15b5f8fc073c1f5ddd280b1e9452b4ccf951736467eda67 not found: ID does not exist" Nov 03 15:46:34.246968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.246874 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn"] Nov 03 15:46:34.259917 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.259886 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f59d26-67b55dbdd7-r4sbn"] Nov 03 15:46:34.517774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.517738 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw"] Nov 03 15:46:34.530052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.530018 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f59d26-5579b6db5f-lz5qw"] Nov 03 15:46:34.681098 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.681065 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c74b1f4-9c5f-4909-bb4e-92181b087fb2" path="/var/lib/kubelet/pods/7c74b1f4-9c5f-4909-bb4e-92181b087fb2/volumes" Nov 03 15:46:34.681510 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:34.681494 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8671d1f-ea5b-4433-af1d-40b53480728b" path="/var/lib/kubelet/pods/b8671d1f-ea5b-4433-af1d-40b53480728b/volumes" Nov 03 15:46:35.265606 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:35.265572 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" Nov 03 15:46:35.316888 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:35.316845 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-maas-api-xks-ca\") pod \"d67ad11f-ae4c-49ca-9763-d2eef45ac27d\" (UID: \"d67ad11f-ae4c-49ca-9763-d2eef45ac27d\") " Nov 03 15:46:35.317127 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:35.317099 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-maas-api-openshift-ca\") pod \"d67ad11f-ae4c-49ca-9763-d2eef45ac27d\" (UID: \"d67ad11f-ae4c-49ca-9763-d2eef45ac27d\") " Nov 03 15:46:35.317245 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:35.317156 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-kube-api-access-gmrnf\" (UniqueName: \"kubernetes.io/projected/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-kube-api-access-gmrnf\") pod \"d67ad11f-ae4c-49ca-9763-d2eef45ac27d\" (UID: \"d67ad11f-ae4c-49ca-9763-d2eef45ac27d\") " Nov 03 15:46:35.317470 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:35.317444 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-maas-api-openshift-ca" pod "d67ad11f-ae4c-49ca-9763-d2eef45ac27d" (UID: "d67ad11f-ae4c-49ca-9763-d2eef45ac27d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:35.318072 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:35.318045 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:35.319321 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:35.319288 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-maas-api-xks-ca" pod "d67ad11f-ae4c-49ca-9763-d2eef45ac27d" (UID: "d67ad11f-ae4c-49ca-9763-d2eef45ac27d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:35.319453 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:35.319438 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-kube-api-access-gmrnf" pod "d67ad11f-ae4c-49ca-9763-d2eef45ac27d" (UID: "d67ad11f-ae4c-49ca-9763-d2eef45ac27d"). InnerVolumeSpecName "kube-api-access-gmrnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:35.418927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:35.418891 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:35.418927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:35.418921 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmrnf\" (UniqueName: \"kubernetes.io/projected/d67ad11f-ae4c-49ca-9763-d2eef45ac27d-kube-api-access-gmrnf\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.101510 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.101474 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" event={"ID":"d67ad11f-ae4c-49ca-9763-d2eef45ac27d","Type":"ContainerDied","Data":"4b92a02684b33893a16a0745d3b592585b928fd0aeb9ae26ea687e29ef8b1e8c"} Nov 03 15:46:36.101510 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.101500 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql" Nov 03 15:46:36.102231 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.101529 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b92a02684b33893a16a0745d3b592585b928fd0aeb9ae26ea687e29ef8b1e8c" Nov 03 15:46:36.194210 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.194163 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" containerID="cri-o://251d68f04ec0c8eca621a49a0232d46d7a29cca8feff72472b7e9e30e823d98a" gracePeriod=45 Nov 03 15:46:36.238468 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.238429 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" containerID="cri-o://d1c7076c686cbd27edfa7b1efe4570cd90f3f04d10973169ec5c6be2a534f41a" gracePeriod=45 Nov 03 15:46:36.535846 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.535813 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:46:36.572153 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.572122 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:46:36.640999 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.640968 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f90c3dc4-9f49-432a-9404-dd55360bcec0-kube-api-access-74jrn\" (UniqueName: \"kubernetes.io/projected/f90c3dc4-9f49-432a-9404-dd55360bcec0-kube-api-access-74jrn\") pod \"f90c3dc4-9f49-432a-9404-dd55360bcec0\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " Nov 03 15:46:36.641201 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.641019 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-home\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-home\") pod \"f90c3dc4-9f49-432a-9404-dd55360bcec0\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " Nov 03 15:46:36.641201 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.641058 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-dshm\") pod \"f90c3dc4-9f49-432a-9404-dd55360bcec0\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " Nov 03 15:46:36.641201 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.641102 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-tmp-dir\") pod \"f90c3dc4-9f49-432a-9404-dd55360bcec0\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " Nov 03 15:46:36.641201 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.641174 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f90c3dc4-9f49-432a-9404-dd55360bcec0-tls-certs\" (UniqueName: \"kubernetes.io/secret/f90c3dc4-9f49-432a-9404-dd55360bcec0-tls-certs\") pod \"f90c3dc4-9f49-432a-9404-dd55360bcec0\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " Nov 03 15:46:36.641440 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.641247 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-model-cache\") pod \"4b83d9a7-0109-4660-aa46-a64c544fcb96\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " Nov 03 15:46:36.641440 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.641282 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-home\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-home\") pod \"4b83d9a7-0109-4660-aa46-a64c544fcb96\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " Nov 03 15:46:36.641440 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.641328 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-dshm\") pod \"4b83d9a7-0109-4660-aa46-a64c544fcb96\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " Nov 03 15:46:36.641440 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.641390 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b83d9a7-0109-4660-aa46-a64c544fcb96-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b83d9a7-0109-4660-aa46-a64c544fcb96-tls-certs\") pod \"4b83d9a7-0109-4660-aa46-a64c544fcb96\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " Nov 03 15:46:36.641665 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.641448 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-model-cache\") pod \"f90c3dc4-9f49-432a-9404-dd55360bcec0\" (UID: \"f90c3dc4-9f49-432a-9404-dd55360bcec0\") " Nov 03 15:46:36.641665 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.641467 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-tmp-dir" pod "f90c3dc4-9f49-432a-9404-dd55360bcec0" (UID: "f90c3dc4-9f49-432a-9404-dd55360bcec0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.641665 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.641550 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b83d9a7-0109-4660-aa46-a64c544fcb96-kube-api-access-hgjcw\" (UniqueName: \"kubernetes.io/projected/4b83d9a7-0109-4660-aa46-a64c544fcb96-kube-api-access-hgjcw\") pod \"4b83d9a7-0109-4660-aa46-a64c544fcb96\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " Nov 03 15:46:36.641665 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.641604 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-tmp-dir\") pod \"4b83d9a7-0109-4660-aa46-a64c544fcb96\" (UID: \"4b83d9a7-0109-4660-aa46-a64c544fcb96\") " Nov 03 15:46:36.641665 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.641653 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-home" pod "f90c3dc4-9f49-432a-9404-dd55360bcec0" (UID: "f90c3dc4-9f49-432a-9404-dd55360bcec0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.642002 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.641799 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-home" pod "4b83d9a7-0109-4660-aa46-a64c544fcb96" (UID: "4b83d9a7-0109-4660-aa46-a64c544fcb96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.642002 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.641897 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-model-cache" pod "4b83d9a7-0109-4660-aa46-a64c544fcb96" (UID: "4b83d9a7-0109-4660-aa46-a64c544fcb96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.642111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.642019 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-model-cache" pod "f90c3dc4-9f49-432a-9404-dd55360bcec0" (UID: "f90c3dc4-9f49-432a-9404-dd55360bcec0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.642264 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.642238 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-tmp-dir" pod "4b83d9a7-0109-4660-aa46-a64c544fcb96" (UID: "4b83d9a7-0109-4660-aa46-a64c544fcb96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.642887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.642781 4290 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-model-cache\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.642887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.642809 4290 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-home\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.642887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.642825 4290 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-model-cache\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.642887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.642837 4290 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-tmp-dir\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.642887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.642850 4290 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-home\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.642887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.642862 4290 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-tmp-dir\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.645650 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.645600 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b83d9a7-0109-4660-aa46-a64c544fcb96-kube-api-access-hgjcw" pod "4b83d9a7-0109-4660-aa46-a64c544fcb96" (UID: "4b83d9a7-0109-4660-aa46-a64c544fcb96"). InnerVolumeSpecName "kube-api-access-hgjcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:36.646033 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.646005 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90c3dc4-9f49-432a-9404-dd55360bcec0-kube-api-access-74jrn" pod "f90c3dc4-9f49-432a-9404-dd55360bcec0" (UID: "f90c3dc4-9f49-432a-9404-dd55360bcec0"). InnerVolumeSpecName "kube-api-access-74jrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:36.646150 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.646059 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-dshm" pod "f90c3dc4-9f49-432a-9404-dd55360bcec0" (UID: "f90c3dc4-9f49-432a-9404-dd55360bcec0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.646329 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.646246 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b83d9a7-0109-4660-aa46-a64c544fcb96-tls-certs" pod "4b83d9a7-0109-4660-aa46-a64c544fcb96" (UID: "4b83d9a7-0109-4660-aa46-a64c544fcb96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:36.646680 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.646636 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-dshm" pod "4b83d9a7-0109-4660-aa46-a64c544fcb96" (UID: "4b83d9a7-0109-4660-aa46-a64c544fcb96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.646930 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.646910 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f90c3dc4-9f49-432a-9404-dd55360bcec0-tls-certs" pod "f90c3dc4-9f49-432a-9404-dd55360bcec0" (UID: "f90c3dc4-9f49-432a-9404-dd55360bcec0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:36.744657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.744573 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgjcw\" (UniqueName: \"kubernetes.io/projected/4b83d9a7-0109-4660-aa46-a64c544fcb96-kube-api-access-hgjcw\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.744657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.744662 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74jrn\" (UniqueName: \"kubernetes.io/projected/f90c3dc4-9f49-432a-9404-dd55360bcec0-kube-api-access-74jrn\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.744978 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.744682 4290 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f90c3dc4-9f49-432a-9404-dd55360bcec0-dshm\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.744978 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.744715 4290 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f90c3dc4-9f49-432a-9404-dd55360bcec0-tls-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.744978 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.744730 4290 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b83d9a7-0109-4660-aa46-a64c544fcb96-dshm\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.744978 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:36.744743 4290 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b83d9a7-0109-4660-aa46-a64c544fcb96-tls-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:37.112434 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.112344 4290 generic.go:356] "Generic (PLEG): container finished" podID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerID="251d68f04ec0c8eca621a49a0232d46d7a29cca8feff72472b7e9e30e823d98a" exitCode=0 Nov 03 15:46:37.112434 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.112386 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" Nov 03 15:46:37.112975 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.112428 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" event={"ID":"4b83d9a7-0109-4660-aa46-a64c544fcb96","Type":"ContainerDied","Data":"251d68f04ec0c8eca621a49a0232d46d7a29cca8feff72472b7e9e30e823d98a"} Nov 03 15:46:37.112975 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.112472 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl" event={"ID":"4b83d9a7-0109-4660-aa46-a64c544fcb96","Type":"ContainerDied","Data":"1ca38ce4ba1012031156a975699792d4ab9d6e311e8432eaa8ded69f6e7ea1fd"} Nov 03 15:46:37.112975 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.112493 4290 scope.go:122] "RemoveContainer" containerID="251d68f04ec0c8eca621a49a0232d46d7a29cca8feff72472b7e9e30e823d98a" Nov 03 15:46:37.115587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.114462 4290 generic.go:356] "Generic (PLEG): container finished" podID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerID="d1c7076c686cbd27edfa7b1efe4570cd90f3f04d10973169ec5c6be2a534f41a" exitCode=0 Nov 03 15:46:37.115587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.114506 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" event={"ID":"f90c3dc4-9f49-432a-9404-dd55360bcec0","Type":"ContainerDied","Data":"d1c7076c686cbd27edfa7b1efe4570cd90f3f04d10973169ec5c6be2a534f41a"} Nov 03 15:46:37.115587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.114560 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" event={"ID":"f90c3dc4-9f49-432a-9404-dd55360bcec0","Type":"ContainerDied","Data":"033b102e148172b3e0ac5e0efcbdc1bb35864402f217cf164bd240624dd7a318"} Nov 03 15:46:37.115587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.114513 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9" Nov 03 15:46:37.124440 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.124420 4290 scope.go:122] "RemoveContainer" containerID="251d68f04ec0c8eca621a49a0232d46d7a29cca8feff72472b7e9e30e823d98a" Nov 03 15:46:37.124765 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:46:37.124746 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"251d68f04ec0c8eca621a49a0232d46d7a29cca8feff72472b7e9e30e823d98a\": container with ID starting with 251d68f04ec0c8eca621a49a0232d46d7a29cca8feff72472b7e9e30e823d98a not found: ID does not exist" containerID="251d68f04ec0c8eca621a49a0232d46d7a29cca8feff72472b7e9e30e823d98a" Nov 03 15:46:37.124818 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.124775 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251d68f04ec0c8eca621a49a0232d46d7a29cca8feff72472b7e9e30e823d98a"} err="failed to get container status \"251d68f04ec0c8eca621a49a0232d46d7a29cca8feff72472b7e9e30e823d98a\": rpc error: code = NotFound desc = could not find container \"251d68f04ec0c8eca621a49a0232d46d7a29cca8feff72472b7e9e30e823d98a\": container with ID starting with 251d68f04ec0c8eca621a49a0232d46d7a29cca8feff72472b7e9e30e823d98a not found: ID does not exist" Nov 03 15:46:37.124818 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.124793 4290 scope.go:122] "RemoveContainer" containerID="d1c7076c686cbd27edfa7b1efe4570cd90f3f04d10973169ec5c6be2a534f41a" Nov 03 15:46:37.135169 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.135147 4290 scope.go:122] "RemoveContainer" containerID="d1c7076c686cbd27edfa7b1efe4570cd90f3f04d10973169ec5c6be2a534f41a" Nov 03 15:46:37.135488 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:46:37.135469 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c7076c686cbd27edfa7b1efe4570cd90f3f04d10973169ec5c6be2a534f41a\": container with ID starting with d1c7076c686cbd27edfa7b1efe4570cd90f3f04d10973169ec5c6be2a534f41a not found: ID does not exist" containerID="d1c7076c686cbd27edfa7b1efe4570cd90f3f04d10973169ec5c6be2a534f41a" Nov 03 15:46:37.135556 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.135496 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c7076c686cbd27edfa7b1efe4570cd90f3f04d10973169ec5c6be2a534f41a"} err="failed to get container status \"d1c7076c686cbd27edfa7b1efe4570cd90f3f04d10973169ec5c6be2a534f41a\": rpc error: code = NotFound desc = could not find container \"d1c7076c686cbd27edfa7b1efe4570cd90f3f04d10973169ec5c6be2a534f41a\": container with ID starting with d1c7076c686cbd27edfa7b1efe4570cd90f3f04d10973169ec5c6be2a534f41a not found: ID does not exist" Nov 03 15:46:37.292805 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.292773 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl"] Nov 03 15:46:37.342124 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.342086 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/distinct-2-w6-f59d26-kserve-d57dbb576-thbvl"] Nov 03 15:46:37.410269 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.410237 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9"] Nov 03 15:46:37.414183 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:37.414156 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/premium-w6-f59d26-kserve-5b4df47dc6-c7pl9"] Nov 03 15:46:38.688421 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:38.688384 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" path="/var/lib/kubelet/pods/4b83d9a7-0109-4660-aa46-a64c544fcb96/volumes" Nov 03 15:46:38.689675 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:38.689651 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" path="/var/lib/kubelet/pods/f90c3dc4-9f49-432a-9404-dd55360bcec0/volumes" Nov 03 15:46:38.711677 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:46:38.711639 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:46:42.676513 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:42.676478 4290 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" secret="" err="secret \"e2e-api-b-90fcca-openshift-default-dockercfg-g7lcc\" not found" Nov 03 15:46:42.715219 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:42.715186 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc"] Nov 03 15:46:42.738963 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:42.738926 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h"] Nov 03 15:46:42.739172 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:42.739148 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" podUID="b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" containerName="istio-proxy" containerID="cri-o://171cb99bc27c60c11a56062dfb0f22637a4c563de812d0c374733e841c667337" gracePeriod=30 Nov 03 15:46:42.743657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:42.743632 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b"] Nov 03 15:46:42.743845 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:42.743825 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" podUID="822b28e8-240d-4cf5-b6de-87900b6c5938" containerName="istio-proxy" containerID="cri-o://8cc669f600652554e32560998f1e2684a971806e755a6a93298512e5bf16c1e7" gracePeriod=30 Nov 03 15:46:43.170654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:43.170595 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" podUID="44a99bae-0003-49ad-99e7-6a53b9d80867" containerName="istio-proxy" containerID="cri-o://5a1af2e6c69280e43f3ee0183c58a3b4c9b037e4eb67884b959a87b7b332b3a1" gracePeriod=30 Nov 03 15:46:43.273949 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:43.273914 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9"] Nov 03 15:46:48.041492 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.041464 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:46:48.044856 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.044833 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:46:48.113577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.113541 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-token\" (UniqueName: \"kubernetes.io/projected/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-token\") pod \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " Nov 03 15:46:48.113577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.113580 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-data\") pod \"822b28e8-240d-4cf5-b6de-87900b6c5938\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " Nov 03 15:46:48.113809 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.113626 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-workload-socket\") pod \"822b28e8-240d-4cf5-b6de-87900b6c5938\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " Nov 03 15:46:48.113809 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.113757 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-podinfo\") pod \"822b28e8-240d-4cf5-b6de-87900b6c5938\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " Nov 03 15:46:48.113905 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.113812 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-workload-socket\") pod \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " Nov 03 15:46:48.113905 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.113882 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822b28e8-240d-4cf5-b6de-87900b6c5938-kube-api-access-8n4d8\" (UniqueName: \"kubernetes.io/projected/822b28e8-240d-4cf5-b6de-87900b6c5938-kube-api-access-8n4d8\") pod \"822b28e8-240d-4cf5-b6de-87900b6c5938\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " Nov 03 15:46:48.113905 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.113895 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-workload-socket" pod "822b28e8-240d-4cf5-b6de-87900b6c5938" (UID: "822b28e8-240d-4cf5-b6de-87900b6c5938"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.114071 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.113937 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-token\" (UniqueName: \"kubernetes.io/projected/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-token\") pod \"822b28e8-240d-4cf5-b6de-87900b6c5938\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " Nov 03 15:46:48.114071 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.113947 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-data" pod "822b28e8-240d-4cf5-b6de-87900b6c5938" (UID: "822b28e8-240d-4cf5-b6de-87900b6c5938"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.114071 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.113979 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-credential-socket\") pod \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " Nov 03 15:46:48.114071 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.114007 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-data\") pod \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " Nov 03 15:46:48.114396 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.114089 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-kube-api-access-frw5h\" (UniqueName: \"kubernetes.io/projected/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-kube-api-access-frw5h\") pod \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " Nov 03 15:46:48.114396 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.114115 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-podinfo\") pod \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " Nov 03 15:46:48.114396 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.114121 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-workload-socket" pod "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" (UID: "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.114396 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.114191 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/822b28e8-240d-4cf5-b6de-87900b6c5938-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822b28e8-240d-4cf5-b6de-87900b6c5938-istiod-ca-cert\") pod \"822b28e8-240d-4cf5-b6de-87900b6c5938\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " Nov 03 15:46:48.114396 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.114253 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-envoy\") pod \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " Nov 03 15:46:48.114396 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.114291 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-data" pod "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" (UID: "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.114743 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.114447 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-workload-certs\") pod \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " Nov 03 15:46:48.114743 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.114468 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-credential-socket" pod "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" (UID: "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.114743 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.114589 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-workload-certs\") pod \"822b28e8-240d-4cf5-b6de-87900b6c5938\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " Nov 03 15:46:48.114743 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.114625 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istiod-ca-cert\") pod \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\" (UID: \"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0\") " Nov 03 15:46:48.114743 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.114665 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-credential-socket\") pod \"822b28e8-240d-4cf5-b6de-87900b6c5938\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " Nov 03 15:46:48.114743 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.114734 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-envoy\") pod \"822b28e8-240d-4cf5-b6de-87900b6c5938\" (UID: \"822b28e8-240d-4cf5-b6de-87900b6c5938\") " Nov 03 15:46:48.115087 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.114770 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822b28e8-240d-4cf5-b6de-87900b6c5938-istiod-ca-cert" pod "822b28e8-240d-4cf5-b6de-87900b6c5938" (UID: "822b28e8-240d-4cf5-b6de-87900b6c5938"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:48.115360 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.115285 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-workload-certs" pod "822b28e8-240d-4cf5-b6de-87900b6c5938" (UID: "822b28e8-240d-4cf5-b6de-87900b6c5938"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.115462 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.115432 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.115462 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.115452 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.115807 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.115465 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822b28e8-240d-4cf5-b6de-87900b6c5938-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.115807 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.115479 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.115807 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.115491 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.115807 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.115507 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.115807 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.115521 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.117011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.116742 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-token" pod "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" (UID: "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:48.117011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.116791 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822b28e8-240d-4cf5-b6de-87900b6c5938-kube-api-access-8n4d8" pod "822b28e8-240d-4cf5-b6de-87900b6c5938" (UID: "822b28e8-240d-4cf5-b6de-87900b6c5938"). InnerVolumeSpecName "kube-api-access-8n4d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:48.117011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.116861 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-podinfo" pod "822b28e8-240d-4cf5-b6de-87900b6c5938" (UID: "822b28e8-240d-4cf5-b6de-87900b6c5938"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:46:48.117011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.116907 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-podinfo" pod "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" (UID: "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:46:48.117011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.117004 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-credential-socket" pod "822b28e8-240d-4cf5-b6de-87900b6c5938" (UID: "822b28e8-240d-4cf5-b6de-87900b6c5938"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.117295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.117078 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-workload-certs" pod "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" (UID: "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.117295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.117117 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istiod-ca-cert" pod "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" (UID: "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:48.117463 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.117443 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-kube-api-access-frw5h" pod "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" (UID: "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0"). InnerVolumeSpecName "kube-api-access-frw5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:48.117636 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.117620 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-envoy" pod "822b28e8-240d-4cf5-b6de-87900b6c5938" (UID: "822b28e8-240d-4cf5-b6de-87900b6c5938"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.118584 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.118558 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-envoy" pod "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" (UID: "b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.118673 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.118600 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-token" pod "822b28e8-240d-4cf5-b6de-87900b6c5938" (UID: "822b28e8-240d-4cf5-b6de-87900b6c5938"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:48.217599 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.217561 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.217599 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.217602 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.217935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.217617 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.217935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.217633 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.217935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.217647 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.217935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.217664 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.217935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.217678 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.217935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.217717 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n4d8\" (UniqueName: \"kubernetes.io/projected/822b28e8-240d-4cf5-b6de-87900b6c5938-kube-api-access-8n4d8\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.217935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.217731 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/822b28e8-240d-4cf5-b6de-87900b6c5938-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.217935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.217746 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frw5h\" (UniqueName: \"kubernetes.io/projected/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-kube-api-access-frw5h\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.217935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.217761 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.230712 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.222126 4290 generic.go:356] "Generic (PLEG): container finished" podID="b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" containerID="171cb99bc27c60c11a56062dfb0f22637a4c563de812d0c374733e841c667337" exitCode=0 Nov 03 15:46:48.230712 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.222463 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" Nov 03 15:46:48.230712 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.223182 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" event={"ID":"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0","Type":"ContainerDied","Data":"171cb99bc27c60c11a56062dfb0f22637a4c563de812d0c374733e841c667337"} Nov 03 15:46:48.230712 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.223245 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h" event={"ID":"b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0","Type":"ContainerDied","Data":"4a3f6525d07ef9cb37d6f60dae758a81e86392afe02e2ae0a06d011eb52ad5e7"} Nov 03 15:46:48.230712 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.223278 4290 scope.go:122] "RemoveContainer" containerID="171cb99bc27c60c11a56062dfb0f22637a4c563de812d0c374733e841c667337" Nov 03 15:46:48.230712 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.225226 4290 generic.go:356] "Generic (PLEG): container finished" podID="822b28e8-240d-4cf5-b6de-87900b6c5938" containerID="8cc669f600652554e32560998f1e2684a971806e755a6a93298512e5bf16c1e7" exitCode=0 Nov 03 15:46:48.230712 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.225289 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" event={"ID":"822b28e8-240d-4cf5-b6de-87900b6c5938","Type":"ContainerDied","Data":"8cc669f600652554e32560998f1e2684a971806e755a6a93298512e5bf16c1e7"} Nov 03 15:46:48.230712 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.225322 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" event={"ID":"822b28e8-240d-4cf5-b6de-87900b6c5938","Type":"ContainerDied","Data":"a202cbe01f9d77810e7db9775287c8f8b568622b27fe356db535732b47d65714"} Nov 03 15:46:48.230712 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.225374 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b" Nov 03 15:46:48.298000 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.297974 4290 scope.go:122] "RemoveContainer" containerID="171cb99bc27c60c11a56062dfb0f22637a4c563de812d0c374733e841c667337" Nov 03 15:46:48.301411 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:46:48.299918 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171cb99bc27c60c11a56062dfb0f22637a4c563de812d0c374733e841c667337\": container with ID starting with 171cb99bc27c60c11a56062dfb0f22637a4c563de812d0c374733e841c667337 not found: ID does not exist" containerID="171cb99bc27c60c11a56062dfb0f22637a4c563de812d0c374733e841c667337" Nov 03 15:46:48.301411 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.299964 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171cb99bc27c60c11a56062dfb0f22637a4c563de812d0c374733e841c667337"} err="failed to get container status \"171cb99bc27c60c11a56062dfb0f22637a4c563de812d0c374733e841c667337\": rpc error: code = NotFound desc = could not find container \"171cb99bc27c60c11a56062dfb0f22637a4c563de812d0c374733e841c667337\": container with ID starting with 171cb99bc27c60c11a56062dfb0f22637a4c563de812d0c374733e841c667337 not found: ID does not exist" Nov 03 15:46:48.301411 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.299989 4290 scope.go:122] "RemoveContainer" containerID="8cc669f600652554e32560998f1e2684a971806e755a6a93298512e5bf16c1e7" Nov 03 15:46:48.311070 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.311042 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n"] Nov 03 15:46:48.325739 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.325571 4290 scope.go:122] "RemoveContainer" containerID="8cc669f600652554e32560998f1e2684a971806e755a6a93298512e5bf16c1e7" Nov 03 15:46:48.326232 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:46:48.326201 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc669f600652554e32560998f1e2684a971806e755a6a93298512e5bf16c1e7\": container with ID starting with 8cc669f600652554e32560998f1e2684a971806e755a6a93298512e5bf16c1e7 not found: ID does not exist" containerID="8cc669f600652554e32560998f1e2684a971806e755a6a93298512e5bf16c1e7" Nov 03 15:46:48.326331 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.326251 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc669f600652554e32560998f1e2684a971806e755a6a93298512e5bf16c1e7"} err="failed to get container status \"8cc669f600652554e32560998f1e2684a971806e755a6a93298512e5bf16c1e7\": rpc error: code = NotFound desc = could not find container \"8cc669f600652554e32560998f1e2684a971806e755a6a93298512e5bf16c1e7\": container with ID starting with 8cc669f600652554e32560998f1e2684a971806e755a6a93298512e5bf16c1e7 not found: ID does not exist" Nov 03 15:46:48.447661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.447628 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h"] Nov 03 15:46:48.454561 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.454531 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-97d6h"] Nov 03 15:46:48.487312 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.487279 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:46:48.586633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.586541 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b"] Nov 03 15:46:48.597984 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.597946 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kqj7b"] Nov 03 15:46:48.623931 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.623900 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-podinfo\") pod \"44a99bae-0003-49ad-99e7-6a53b9d80867\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " Nov 03 15:46:48.624116 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.623997 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-token\" (UniqueName: \"kubernetes.io/projected/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-token\") pod \"44a99bae-0003-49ad-99e7-6a53b9d80867\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " Nov 03 15:46:48.624116 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.624063 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a99bae-0003-49ad-99e7-6a53b9d80867-kube-api-access-rqk2d\" (UniqueName: \"kubernetes.io/projected/44a99bae-0003-49ad-99e7-6a53b9d80867-kube-api-access-rqk2d\") pod \"44a99bae-0003-49ad-99e7-6a53b9d80867\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " Nov 03 15:46:48.624116 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.624106 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a99bae-0003-49ad-99e7-6a53b9d80867-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a99bae-0003-49ad-99e7-6a53b9d80867-istiod-ca-cert\") pod \"44a99bae-0003-49ad-99e7-6a53b9d80867\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " Nov 03 15:46:48.624293 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.624156 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-workload-certs\") pod \"44a99bae-0003-49ad-99e7-6a53b9d80867\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " Nov 03 15:46:48.624293 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.624192 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-data\") pod \"44a99bae-0003-49ad-99e7-6a53b9d80867\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " Nov 03 15:46:48.624293 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.624266 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-credential-socket\") pod \"44a99bae-0003-49ad-99e7-6a53b9d80867\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " Nov 03 15:46:48.624454 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.624382 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-envoy\") pod \"44a99bae-0003-49ad-99e7-6a53b9d80867\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " Nov 03 15:46:48.624454 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.624447 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-workload-socket\") pod \"44a99bae-0003-49ad-99e7-6a53b9d80867\" (UID: \"44a99bae-0003-49ad-99e7-6a53b9d80867\") " Nov 03 15:46:48.624586 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.624559 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a99bae-0003-49ad-99e7-6a53b9d80867-istiod-ca-cert" pod "44a99bae-0003-49ad-99e7-6a53b9d80867" (UID: "44a99bae-0003-49ad-99e7-6a53b9d80867"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:48.624833 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.624771 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-credential-socket" pod "44a99bae-0003-49ad-99e7-6a53b9d80867" (UID: "44a99bae-0003-49ad-99e7-6a53b9d80867"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.625041 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.624855 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-data" pod "44a99bae-0003-49ad-99e7-6a53b9d80867" (UID: "44a99bae-0003-49ad-99e7-6a53b9d80867"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.625041 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.625006 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a99bae-0003-49ad-99e7-6a53b9d80867-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.625041 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.625025 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.625223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.625042 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.625223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.625084 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-workload-certs" pod "44a99bae-0003-49ad-99e7-6a53b9d80867" (UID: "44a99bae-0003-49ad-99e7-6a53b9d80867"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.625223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.625140 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-workload-socket" pod "44a99bae-0003-49ad-99e7-6a53b9d80867" (UID: "44a99bae-0003-49ad-99e7-6a53b9d80867"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.626476 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.626452 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-podinfo" pod "44a99bae-0003-49ad-99e7-6a53b9d80867" (UID: "44a99bae-0003-49ad-99e7-6a53b9d80867"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:46:48.626771 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.626749 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-envoy" pod "44a99bae-0003-49ad-99e7-6a53b9d80867" (UID: "44a99bae-0003-49ad-99e7-6a53b9d80867"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.626971 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.626946 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a99bae-0003-49ad-99e7-6a53b9d80867-kube-api-access-rqk2d" pod "44a99bae-0003-49ad-99e7-6a53b9d80867" (UID: "44a99bae-0003-49ad-99e7-6a53b9d80867"). InnerVolumeSpecName "kube-api-access-rqk2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:48.626971 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.626958 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-token" pod "44a99bae-0003-49ad-99e7-6a53b9d80867" (UID: "44a99bae-0003-49ad-99e7-6a53b9d80867"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:48.678845 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.678808 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822b28e8-240d-4cf5-b6de-87900b6c5938" path="/var/lib/kubelet/pods/822b28e8-240d-4cf5-b6de-87900b6c5938/volumes" Nov 03 15:46:48.679391 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.679369 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" path="/var/lib/kubelet/pods/b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0/volumes" Nov 03 15:46:48.726430 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.726390 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqk2d\" (UniqueName: \"kubernetes.io/projected/44a99bae-0003-49ad-99e7-6a53b9d80867-kube-api-access-rqk2d\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.726430 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.726425 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.726430 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.726435 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.726668 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.726449 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a99bae-0003-49ad-99e7-6a53b9d80867-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.726668 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.726460 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.726668 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:48.726468 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a99bae-0003-49ad-99e7-6a53b9d80867-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:49.236153 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:49.236120 4290 generic.go:356] "Generic (PLEG): container finished" podID="44a99bae-0003-49ad-99e7-6a53b9d80867" containerID="5a1af2e6c69280e43f3ee0183c58a3b4c9b037e4eb67884b959a87b7b332b3a1" exitCode=0 Nov 03 15:46:49.236624 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:49.236169 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" Nov 03 15:46:49.236624 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:49.236205 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" event={"ID":"44a99bae-0003-49ad-99e7-6a53b9d80867","Type":"ContainerDied","Data":"5a1af2e6c69280e43f3ee0183c58a3b4c9b037e4eb67884b959a87b7b332b3a1"} Nov 03 15:46:49.236624 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:49.236253 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc" event={"ID":"44a99bae-0003-49ad-99e7-6a53b9d80867","Type":"ContainerDied","Data":"032f134ec341ae5ee0dd4556e2bb9e25ae33c602b9b2aa87153740b4582599e9"} Nov 03 15:46:49.236624 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:49.236281 4290 scope.go:122] "RemoveContainer" containerID="5a1af2e6c69280e43f3ee0183c58a3b4c9b037e4eb67884b959a87b7b332b3a1" Nov 03 15:46:49.249975 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:49.249950 4290 scope.go:122] "RemoveContainer" containerID="5a1af2e6c69280e43f3ee0183c58a3b4c9b037e4eb67884b959a87b7b332b3a1" Nov 03 15:46:49.250311 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:46:49.250290 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1af2e6c69280e43f3ee0183c58a3b4c9b037e4eb67884b959a87b7b332b3a1\": container with ID starting with 5a1af2e6c69280e43f3ee0183c58a3b4c9b037e4eb67884b959a87b7b332b3a1 not found: ID does not exist" containerID="5a1af2e6c69280e43f3ee0183c58a3b4c9b037e4eb67884b959a87b7b332b3a1" Nov 03 15:46:49.250389 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:49.250319 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1af2e6c69280e43f3ee0183c58a3b4c9b037e4eb67884b959a87b7b332b3a1"} err="failed to get container status \"5a1af2e6c69280e43f3ee0183c58a3b4c9b037e4eb67884b959a87b7b332b3a1\": rpc error: code = NotFound desc = could not find container \"5a1af2e6c69280e43f3ee0183c58a3b4c9b037e4eb67884b959a87b7b332b3a1\": container with ID starting with 5a1af2e6c69280e43f3ee0183c58a3b4c9b037e4eb67884b959a87b7b332b3a1 not found: ID does not exist" Nov 03 15:46:49.417440 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:49.417400 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc"] Nov 03 15:46:49.431227 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:49.431196 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-m8sxc"] Nov 03 15:46:50.497060 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497025 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n"] Nov 03 15:46:50.497629 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497615 4290 cpu_manager.go:414] "containerMap: removing container" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" Nov 03 15:46:50.497674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497630 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" Nov 03 15:46:50.497674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497646 4290 cpu_manager.go:414] "containerMap: removing container" podUID="44a99bae-0003-49ad-99e7-6a53b9d80867" containerName="istio-proxy" Nov 03 15:46:50.497674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497651 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a99bae-0003-49ad-99e7-6a53b9d80867" containerName="istio-proxy" Nov 03 15:46:50.497674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497659 4290 cpu_manager.go:414] "containerMap: removing container" podUID="b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" containerName="istio-proxy" Nov 03 15:46:50.497674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497664 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" containerName="istio-proxy" Nov 03 15:46:50.497674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497671 4290 cpu_manager.go:414] "containerMap: removing container" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" Nov 03 15:46:50.497674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497676 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497683 4290 cpu_manager.go:414] "containerMap: removing container" podUID="b8671d1f-ea5b-4433-af1d-40b53480728b" containerName="payload-processing" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497711 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8671d1f-ea5b-4433-af1d-40b53480728b" containerName="payload-processing" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497756 4290 cpu_manager.go:414] "containerMap: removing container" podUID="d67ad11f-ae4c-49ca-9763-d2eef45ac27d" containerName="revoke-keys" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497764 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67ad11f-ae4c-49ca-9763-d2eef45ac27d" containerName="revoke-keys" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497776 4290 cpu_manager.go:414] "containerMap: removing container" podUID="7c74b1f4-9c5f-4909-bb4e-92181b087fb2" containerName="payload-pre-processing" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497780 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c74b1f4-9c5f-4909-bb4e-92181b087fb2" containerName="payload-pre-processing" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497791 4290 cpu_manager.go:414] "containerMap: removing container" podUID="822b28e8-240d-4cf5-b6de-87900b6c5938" containerName="istio-proxy" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497796 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822b28e8-240d-4cf5-b6de-87900b6c5938" containerName="istio-proxy" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497892 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67ad11f-ae4c-49ca-9763-d2eef45ac27d" containerName="revoke-keys" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497902 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c74b1f4-9c5f-4909-bb4e-92181b087fb2" containerName="payload-pre-processing" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497910 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="822b28e8-240d-4cf5-b6de-87900b6c5938" containerName="istio-proxy" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497917 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="f90c3dc4-9f49-432a-9404-dd55360bcec0" containerName="main" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497923 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a99bae-0003-49ad-99e7-6a53b9d80867" containerName="istio-proxy" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497928 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bf8c9c-2a0e-4ab1-ab0f-b229ccc4b9e0" containerName="istio-proxy" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497934 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b83d9a7-0109-4660-aa46-a64c544fcb96" containerName="main" Nov 03 15:46:50.497932 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.497939 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8671d1f-ea5b-4433-af1d-40b53480728b" containerName="payload-processing" Nov 03 15:46:50.502904 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.502879 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" Nov 03 15:46:50.616362 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.616331 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:46:50.616508 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.616359 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:46:50.620642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.620616 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n"] Nov 03 15:46:50.652523 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.652491 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05edca37-9fff-4f66-a2bd-a73c4ad46246-maas-api-xks-ca\") pod \"maas-api-revoke-sub-859cf33f-vtp6n\" (UID: \"05edca37-9fff-4f66-a2bd-a73c4ad46246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" Nov 03 15:46:50.652743 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.652717 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05edca37-9fff-4f66-a2bd-a73c4ad46246-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-859cf33f-vtp6n\" (UID: \"05edca37-9fff-4f66-a2bd-a73c4ad46246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" Nov 03 15:46:50.652808 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.652789 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhg4v\" (UniqueName: \"kubernetes.io/projected/05edca37-9fff-4f66-a2bd-a73c4ad46246-kube-api-access-qhg4v\") pod \"maas-api-revoke-sub-859cf33f-vtp6n\" (UID: \"05edca37-9fff-4f66-a2bd-a73c4ad46246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" Nov 03 15:46:50.655450 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.655386 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt"] Nov 03 15:46:50.681933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.681900 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a99bae-0003-49ad-99e7-6a53b9d80867" path="/var/lib/kubelet/pods/44a99bae-0003-49ad-99e7-6a53b9d80867/volumes" Nov 03 15:46:50.753845 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.753768 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05edca37-9fff-4f66-a2bd-a73c4ad46246-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-859cf33f-vtp6n\" (UID: \"05edca37-9fff-4f66-a2bd-a73c4ad46246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" Nov 03 15:46:50.753845 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.753816 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhg4v\" (UniqueName: \"kubernetes.io/projected/05edca37-9fff-4f66-a2bd-a73c4ad46246-kube-api-access-qhg4v\") pod \"maas-api-revoke-sub-859cf33f-vtp6n\" (UID: \"05edca37-9fff-4f66-a2bd-a73c4ad46246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" Nov 03 15:46:50.754079 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.754050 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05edca37-9fff-4f66-a2bd-a73c4ad46246-maas-api-xks-ca\") pod \"maas-api-revoke-sub-859cf33f-vtp6n\" (UID: \"05edca37-9fff-4f66-a2bd-a73c4ad46246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" Nov 03 15:46:50.754597 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.754571 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05edca37-9fff-4f66-a2bd-a73c4ad46246-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-859cf33f-vtp6n\" (UID: \"05edca37-9fff-4f66-a2bd-a73c4ad46246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" Nov 03 15:46:50.756782 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.756737 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05edca37-9fff-4f66-a2bd-a73c4ad46246-maas-api-xks-ca\") pod \"maas-api-revoke-sub-859cf33f-vtp6n\" (UID: \"05edca37-9fff-4f66-a2bd-a73c4ad46246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" Nov 03 15:46:50.817734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.817662 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhg4v\" (UniqueName: \"kubernetes.io/projected/05edca37-9fff-4f66-a2bd-a73c4ad46246-kube-api-access-qhg4v\") pod \"maas-api-revoke-sub-859cf33f-vtp6n\" (UID: \"05edca37-9fff-4f66-a2bd-a73c4ad46246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" Nov 03 15:46:50.817933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.817914 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" Nov 03 15:46:50.860257 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.860217 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9"] Nov 03 15:46:50.860733 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:50.860681 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" podUID="4fbf376d-4692-4f6c-9b1b-c571c2e44a2b" containerName="maas-api" containerID="cri-o://ce869874a9ca27e202cd44490ed67c0a34fc1f446cb89ef5f2c9f276d57e628f" gracePeriod=30 Nov 03 15:46:51.191914 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.191881 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:46:51.238627 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.238598 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n"] Nov 03 15:46:51.238774 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:46:51.238742 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05edca37_9fff_4f66_a2bd_a73c4ad46246.slice/crio-b13935ce73477c710462296eadd7fe541f8d1e359887a78a645349b141eee8ca WatchSource:0}: Error finding container b13935ce73477c710462296eadd7fe541f8d1e359887a78a645349b141eee8ca: Status 404 returned error can't find the container with id b13935ce73477c710462296eadd7fe541f8d1e359887a78a645349b141eee8ca Nov 03 15:46:51.261308 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.261203 4290 generic.go:356] "Generic (PLEG): container finished" podID="4fbf376d-4692-4f6c-9b1b-c571c2e44a2b" containerID="ce869874a9ca27e202cd44490ed67c0a34fc1f446cb89ef5f2c9f276d57e628f" exitCode=0 Nov 03 15:46:51.261308 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.261262 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" Nov 03 15:46:51.261308 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.261285 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" event={"ID":"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b","Type":"ContainerDied","Data":"ce869874a9ca27e202cd44490ed67c0a34fc1f446cb89ef5f2c9f276d57e628f"} Nov 03 15:46:51.261501 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.261348 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9" event={"ID":"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b","Type":"ContainerDied","Data":"6e75bdcbe670cb13044d78fc4309c5aac5d512a3cef3540098081da490296f0e"} Nov 03 15:46:51.261501 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.261381 4290 scope.go:122] "RemoveContainer" containerID="ce869874a9ca27e202cd44490ed67c0a34fc1f446cb89ef5f2c9f276d57e628f" Nov 03 15:46:51.264321 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.264297 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" event={"ID":"05edca37-9fff-4f66-a2bd-a73c4ad46246","Type":"ContainerStarted","Data":"b13935ce73477c710462296eadd7fe541f8d1e359887a78a645349b141eee8ca"} Nov 03 15:46:51.278281 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.278258 4290 scope.go:122] "RemoveContainer" containerID="ce869874a9ca27e202cd44490ed67c0a34fc1f446cb89ef5f2c9f276d57e628f" Nov 03 15:46:51.278620 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:46:51.278598 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce869874a9ca27e202cd44490ed67c0a34fc1f446cb89ef5f2c9f276d57e628f\": container with ID starting with ce869874a9ca27e202cd44490ed67c0a34fc1f446cb89ef5f2c9f276d57e628f not found: ID does not exist" containerID="ce869874a9ca27e202cd44490ed67c0a34fc1f446cb89ef5f2c9f276d57e628f" Nov 03 15:46:51.278681 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.278628 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce869874a9ca27e202cd44490ed67c0a34fc1f446cb89ef5f2c9f276d57e628f"} err="failed to get container status \"ce869874a9ca27e202cd44490ed67c0a34fc1f446cb89ef5f2c9f276d57e628f\": rpc error: code = NotFound desc = could not find container \"ce869874a9ca27e202cd44490ed67c0a34fc1f446cb89ef5f2c9f276d57e628f\": container with ID starting with ce869874a9ca27e202cd44490ed67c0a34fc1f446cb89ef5f2c9f276d57e628f not found: ID does not exist" Nov 03 15:46:51.290329 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.290302 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-maas-api-tls\") pod \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " Nov 03 15:46:51.290478 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.290344 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-openshift-service-ca\") pod \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " Nov 03 15:46:51.290555 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.290500 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-metrics-certs\") pod \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " Nov 03 15:46:51.290615 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.290553 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-kube-api-access-w2nmj\" (UniqueName: \"kubernetes.io/projected/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-kube-api-access-w2nmj\") pod \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\" (UID: \"4fbf376d-4692-4f6c-9b1b-c571c2e44a2b\") " Nov 03 15:46:51.290798 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.290774 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-openshift-service-ca" pod "4fbf376d-4692-4f6c-9b1b-c571c2e44a2b" (UID: "4fbf376d-4692-4f6c-9b1b-c571c2e44a2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:51.291003 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.290986 4290 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-openshift-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:51.292431 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.292404 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-maas-api-tls" pod "4fbf376d-4692-4f6c-9b1b-c571c2e44a2b" (UID: "4fbf376d-4692-4f6c-9b1b-c571c2e44a2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:51.292847 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.292824 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-kube-api-access-w2nmj" pod "4fbf376d-4692-4f6c-9b1b-c571c2e44a2b" (UID: "4fbf376d-4692-4f6c-9b1b-c571c2e44a2b"). InnerVolumeSpecName "kube-api-access-w2nmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:51.292847 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.292837 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-metrics-certs" pod "4fbf376d-4692-4f6c-9b1b-c571c2e44a2b" (UID: "4fbf376d-4692-4f6c-9b1b-c571c2e44a2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:51.391910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.391874 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-maas-api-tls\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:51.391910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.391903 4290 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-metrics-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:51.391910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.391914 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2nmj\" (UniqueName: \"kubernetes.io/projected/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b-kube-api-access-w2nmj\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:51.738186 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.738144 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9"] Nov 03 15:46:51.745946 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:51.745914 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9855ff-7dd4584579-dg6l9"] Nov 03 15:46:52.332561 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:52.332520 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" event={"ID":"05edca37-9fff-4f66-a2bd-a73c4ad46246","Type":"ContainerStarted","Data":"6a0213cbbbe4cb8a7d372da46b34fafa109cdb08ab13fbf106bfe760da2e934b"} Nov 03 15:46:52.456510 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:52.456449 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" podStartSLOduration=2.456427118 podStartE2EDuration="2.456427118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:46:52.451353673 +0000 UTC m=+2319.882311479" watchObservedRunningTime="2026-11-03 15:46:52.456427118 +0000 UTC m=+2319.887384927" Nov 03 15:46:52.679734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:52.679676 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fbf376d-4692-4f6c-9b1b-c571c2e44a2b" path="/var/lib/kubelet/pods/4fbf376d-4692-4f6c-9b1b-c571c2e44a2b/volumes" Nov 03 15:46:53.344544 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:53.344512 4290 generic.go:356] "Generic (PLEG): container finished" podID="05edca37-9fff-4f66-a2bd-a73c4ad46246" containerID="6a0213cbbbe4cb8a7d372da46b34fafa109cdb08ab13fbf106bfe760da2e934b" exitCode=0 Nov 03 15:46:53.344997 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:53.344611 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" event={"ID":"05edca37-9fff-4f66-a2bd-a73c4ad46246","Type":"ContainerDied","Data":"6a0213cbbbe4cb8a7d372da46b34fafa109cdb08ab13fbf106bfe760da2e934b"} Nov 03 15:46:54.524914 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:54.524886 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" Nov 03 15:46:54.543990 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:54.543961 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05edca37-9fff-4f66-a2bd-a73c4ad46246-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05edca37-9fff-4f66-a2bd-a73c4ad46246-maas-api-xks-ca\") pod \"05edca37-9fff-4f66-a2bd-a73c4ad46246\" (UID: \"05edca37-9fff-4f66-a2bd-a73c4ad46246\") " Nov 03 15:46:54.544166 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:54.544099 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05edca37-9fff-4f66-a2bd-a73c4ad46246-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05edca37-9fff-4f66-a2bd-a73c4ad46246-maas-api-openshift-ca\") pod \"05edca37-9fff-4f66-a2bd-a73c4ad46246\" (UID: \"05edca37-9fff-4f66-a2bd-a73c4ad46246\") " Nov 03 15:46:54.544166 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:54.544157 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05edca37-9fff-4f66-a2bd-a73c4ad46246-kube-api-access-qhg4v\" (UniqueName: \"kubernetes.io/projected/05edca37-9fff-4f66-a2bd-a73c4ad46246-kube-api-access-qhg4v\") pod \"05edca37-9fff-4f66-a2bd-a73c4ad46246\" (UID: \"05edca37-9fff-4f66-a2bd-a73c4ad46246\") " Nov 03 15:46:54.544461 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:54.544421 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05edca37-9fff-4f66-a2bd-a73c4ad46246-maas-api-openshift-ca" pod "05edca37-9fff-4f66-a2bd-a73c4ad46246" (UID: "05edca37-9fff-4f66-a2bd-a73c4ad46246"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:54.544592 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:54.544577 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05edca37-9fff-4f66-a2bd-a73c4ad46246-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:54.546438 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:54.546413 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05edca37-9fff-4f66-a2bd-a73c4ad46246-maas-api-xks-ca" pod "05edca37-9fff-4f66-a2bd-a73c4ad46246" (UID: "05edca37-9fff-4f66-a2bd-a73c4ad46246"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:54.546539 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:54.546479 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05edca37-9fff-4f66-a2bd-a73c4ad46246-kube-api-access-qhg4v" pod "05edca37-9fff-4f66-a2bd-a73c4ad46246" (UID: "05edca37-9fff-4f66-a2bd-a73c4ad46246"). InnerVolumeSpecName "kube-api-access-qhg4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:54.645734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:54.645657 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhg4v\" (UniqueName: \"kubernetes.io/projected/05edca37-9fff-4f66-a2bd-a73c4ad46246-kube-api-access-qhg4v\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:54.645734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:54.645735 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05edca37-9fff-4f66-a2bd-a73c4ad46246-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:55.346621 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:55.346587 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t"] Nov 03 15:46:55.363399 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:55.363368 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" Nov 03 15:46:55.363535 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:55.363408 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n" event={"ID":"05edca37-9fff-4f66-a2bd-a73c4ad46246","Type":"ContainerDied","Data":"b13935ce73477c710462296eadd7fe541f8d1e359887a78a645349b141eee8ca"} Nov 03 15:46:55.363535 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:55.363455 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b13935ce73477c710462296eadd7fe541f8d1e359887a78a645349b141eee8ca" Nov 03 15:46:55.454628 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:55.454583 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft"] Nov 03 15:46:55.656625 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:55.656581 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" podUID="76e830bb-3e77-42b6-9d03-d41e24955b37" containerName="payload-pre-processing" containerID="cri-o://a9575c81007505760bcb470892761d230d58ebe5621e4581bfb881ac6312c156" gracePeriod=25 Nov 03 15:46:58.301876 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.301825 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" containerID="cri-o://f06fb4bbb4d8f9d29cf19ee825fccf477f23c22a620d0c8f4fa3cdcc582a8abc" gracePeriod=45 Nov 03 15:46:58.583483 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.583455 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:46:58.606079 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.606046 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-dshm\") pod \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " Nov 03 15:46:58.606245 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.606112 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-kube-api-access-jrmt5\" (UniqueName: \"kubernetes.io/projected/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-kube-api-access-jrmt5\") pod \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " Nov 03 15:46:58.606245 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.606145 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-home\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-home\") pod \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " Nov 03 15:46:58.606245 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.606243 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-tmp-dir\") pod \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " Nov 03 15:46:58.606419 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.606322 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-tls-certs\") pod \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " Nov 03 15:46:58.606419 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.606342 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-model-cache\") pod \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\" (UID: \"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3\") " Nov 03 15:46:58.606522 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.606453 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-home" pod "c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" (UID: "c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:58.606578 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.606548 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-tmp-dir" pod "c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" (UID: "c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:58.606661 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.606632 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-model-cache" pod "c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" (UID: "c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:58.606881 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.606863 4290 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-home\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.606951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.606886 4290 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-tmp-dir\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.606951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.606901 4290 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-model-cache\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.608464 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.608445 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-dshm" pod "c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" (UID: "c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:58.608633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.608605 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-kube-api-access-jrmt5" pod "c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" (UID: "c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3"). InnerVolumeSpecName "kube-api-access-jrmt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:58.608690 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.608636 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-tls-certs" pod "c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" (UID: "c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:58.708584 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.708555 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrmt5\" (UniqueName: \"kubernetes.io/projected/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-kube-api-access-jrmt5\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.708584 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.708582 4290 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-tls-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.708756 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:58.708592 4290 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3-dshm\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:59.395353 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:59.395318 4290 generic.go:356] "Generic (PLEG): container finished" podID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerID="f06fb4bbb4d8f9d29cf19ee825fccf477f23c22a620d0c8f4fa3cdcc582a8abc" exitCode=0 Nov 03 15:46:59.395858 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:59.395366 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" Nov 03 15:46:59.395858 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:59.395401 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" event={"ID":"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3","Type":"ContainerDied","Data":"f06fb4bbb4d8f9d29cf19ee825fccf477f23c22a620d0c8f4fa3cdcc582a8abc"} Nov 03 15:46:59.395858 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:59.395451 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9" event={"ID":"c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3","Type":"ContainerDied","Data":"f263bd25f117b48392ea9595482394fcd398570bcd9989c91d7fdde9715b3f02"} Nov 03 15:46:59.395858 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:59.395473 4290 scope.go:122] "RemoveContainer" containerID="f06fb4bbb4d8f9d29cf19ee825fccf477f23c22a620d0c8f4fa3cdcc582a8abc" Nov 03 15:46:59.406951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:59.406930 4290 scope.go:122] "RemoveContainer" containerID="f06fb4bbb4d8f9d29cf19ee825fccf477f23c22a620d0c8f4fa3cdcc582a8abc" Nov 03 15:46:59.407222 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:46:59.407202 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06fb4bbb4d8f9d29cf19ee825fccf477f23c22a620d0c8f4fa3cdcc582a8abc\": container with ID starting with f06fb4bbb4d8f9d29cf19ee825fccf477f23c22a620d0c8f4fa3cdcc582a8abc not found: ID does not exist" containerID="f06fb4bbb4d8f9d29cf19ee825fccf477f23c22a620d0c8f4fa3cdcc582a8abc" Nov 03 15:46:59.407264 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:59.407231 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06fb4bbb4d8f9d29cf19ee825fccf477f23c22a620d0c8f4fa3cdcc582a8abc"} err="failed to get container status \"f06fb4bbb4d8f9d29cf19ee825fccf477f23c22a620d0c8f4fa3cdcc582a8abc\": rpc error: code = NotFound desc = could not find container \"f06fb4bbb4d8f9d29cf19ee825fccf477f23c22a620d0c8f4fa3cdcc582a8abc\": container with ID starting with f06fb4bbb4d8f9d29cf19ee825fccf477f23c22a620d0c8f4fa3cdcc582a8abc not found: ID does not exist" Nov 03 15:46:59.591386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:59.591355 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9"] Nov 03 15:46:59.640934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:46:59.640895 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/distinct-w0-e62ff3-kserve-69fcf7ddbf-5htb9"] Nov 03 15:47:00.346885 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:00.346835 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" podUID="ad3f085b-e44a-4f7a-a92e-065c19adf7d8" containerName="payload-processing" containerID="cri-o://2eed4a4835c16f79ef0d8917c228f7939821e72dd69278a6a33230da3d11288e" gracePeriod=25 Nov 03 15:47:00.455474 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:00.455423 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" podUID="5ce34d51-acb2-484d-acee-275b15ec6532" containerName="payload-pre-processing" containerID="cri-o://171fd0f269122493c3e40d6c80e6d504bc8d19d4f5439d998e5bfa112cf75c37" gracePeriod=25 Nov 03 15:47:00.683149 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:00.683109 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" path="/var/lib/kubelet/pods/c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3/volumes" Nov 03 15:47:01.451752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:01.451682 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" podUID="76e830bb-3e77-42b6-9d03-d41e24955b37" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:47:02.301073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:02.301030 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh"] Nov 03 15:47:02.301545 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:02.301238 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" podUID="afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" containerName="istio-proxy" containerID="cri-o://321dfb447dbbfe54fa34c09ebb256c6374f30712ab8cc0c712b5cd25dc1334e8" gracePeriod=30 Nov 03 15:47:03.341776 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.341723 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" containerID="cri-o://a15ed5bab533a29f8fc286a0cbe6b43a442ec5ba2e99342e72054484cd97a793" gracePeriod=45 Nov 03 15:47:03.619020 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.618994 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:47:03.673786 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.673748 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-model-cache\") pod \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " Nov 03 15:47:03.673963 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.673796 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-tmp-dir\") pod \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " Nov 03 15:47:03.673963 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.673821 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17b58cf9-2cff-4648-86c1-0b1d09acacdd-tls-certs\" (UniqueName: \"kubernetes.io/secret/17b58cf9-2cff-4648-86c1-0b1d09acacdd-tls-certs\") pod \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " Nov 03 15:47:03.673963 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.673845 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-home\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-home\") pod \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " Nov 03 15:47:03.673963 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.673886 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b58cf9-2cff-4648-86c1-0b1d09acacdd-kube-api-access-9k7r2\" (UniqueName: \"kubernetes.io/projected/17b58cf9-2cff-4648-86c1-0b1d09acacdd-kube-api-access-9k7r2\") pod \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " Nov 03 15:47:03.674137 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.674030 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-model-cache" pod "17b58cf9-2cff-4648-86c1-0b1d09acacdd" (UID: "17b58cf9-2cff-4648-86c1-0b1d09acacdd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:03.674137 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.674035 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-dshm\") pod \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\" (UID: \"17b58cf9-2cff-4648-86c1-0b1d09acacdd\") " Nov 03 15:47:03.674137 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.674100 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-tmp-dir" pod "17b58cf9-2cff-4648-86c1-0b1d09acacdd" (UID: "17b58cf9-2cff-4648-86c1-0b1d09acacdd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:03.674237 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.674171 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-home" pod "17b58cf9-2cff-4648-86c1-0b1d09acacdd" (UID: "17b58cf9-2cff-4648-86c1-0b1d09acacdd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:03.674642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.674619 4290 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-tmp-dir\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.674850 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.674827 4290 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-home\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.674850 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.674850 4290 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-model-cache\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.676406 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.676375 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-dshm" pod "17b58cf9-2cff-4648-86c1-0b1d09acacdd" (UID: "17b58cf9-2cff-4648-86c1-0b1d09acacdd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:03.676506 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.676472 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b58cf9-2cff-4648-86c1-0b1d09acacdd-kube-api-access-9k7r2" pod "17b58cf9-2cff-4648-86c1-0b1d09acacdd" (UID: "17b58cf9-2cff-4648-86c1-0b1d09acacdd"). InnerVolumeSpecName "kube-api-access-9k7r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:03.676573 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.676540 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17b58cf9-2cff-4648-86c1-0b1d09acacdd-tls-certs" pod "17b58cf9-2cff-4648-86c1-0b1d09acacdd" (UID: "17b58cf9-2cff-4648-86c1-0b1d09acacdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:47:03.777070 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.777028 4290 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b58cf9-2cff-4648-86c1-0b1d09acacdd-tls-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.777253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.777068 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k7r2\" (UniqueName: \"kubernetes.io/projected/17b58cf9-2cff-4648-86c1-0b1d09acacdd-kube-api-access-9k7r2\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.777253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.777103 4290 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b58cf9-2cff-4648-86c1-0b1d09acacdd-dshm\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.809941 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.809908 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs"] Nov 03 15:47:03.810506 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.810493 4290 cpu_manager.go:414] "containerMap: removing container" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" Nov 03 15:47:03.810546 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.810508 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" Nov 03 15:47:03.810546 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.810527 4290 cpu_manager.go:414] "containerMap: removing container" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" Nov 03 15:47:03.810546 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.810532 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" Nov 03 15:47:03.810546 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.810545 4290 cpu_manager.go:414] "containerMap: removing container" podUID="05edca37-9fff-4f66-a2bd-a73c4ad46246" containerName="revoke-keys" Nov 03 15:47:03.810674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.810550 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05edca37-9fff-4f66-a2bd-a73c4ad46246" containerName="revoke-keys" Nov 03 15:47:03.810674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.810576 4290 cpu_manager.go:414] "containerMap: removing container" podUID="4fbf376d-4692-4f6c-9b1b-c571c2e44a2b" containerName="maas-api" Nov 03 15:47:03.810674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.810582 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fbf376d-4692-4f6c-9b1b-c571c2e44a2b" containerName="maas-api" Nov 03 15:47:03.810674 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.810668 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbf376d-4692-4f6c-9b1b-c571c2e44a2b" containerName="maas-api" Nov 03 15:47:03.810830 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.810678 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6203a8f-8dc0-4b4b-8d1f-eb6adb071ae3" containerName="main" Nov 03 15:47:03.810830 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.810687 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerName="main" Nov 03 15:47:03.810830 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.810694 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="05edca37-9fff-4f66-a2bd-a73c4ad46246" containerName="revoke-keys" Nov 03 15:47:03.814514 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.814494 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" Nov 03 15:47:03.872265 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.872222 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs"] Nov 03 15:47:03.873654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.873630 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:47:03.873854 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.873631 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:47:03.878956 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.878928 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c405c767-2w5hs\" (UID: \"14cc323b-4437-4c76-adf8-dc2a7dbd4da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" Nov 03 15:47:03.879166 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.879138 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqxrr\" (UniqueName: \"kubernetes.io/projected/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-kube-api-access-sqxrr\") pod \"maas-api-revoke-sub-c405c767-2w5hs\" (UID: \"14cc323b-4437-4c76-adf8-dc2a7dbd4da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" Nov 03 15:47:03.879246 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.879196 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c405c767-2w5hs\" (UID: \"14cc323b-4437-4c76-adf8-dc2a7dbd4da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" Nov 03 15:47:03.925081 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.925046 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt"] Nov 03 15:47:03.925280 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.925258 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" podUID="fbb5aae0-bbca-4f1f-83c3-875d0ffd9001" containerName="maas-api" containerID="cri-o://c9f99df6f375629d00e5e0636c703b74935574979c5abbb36e44333b9d112162" gracePeriod=30 Nov 03 15:47:03.980518 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.980478 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c405c767-2w5hs\" (UID: \"14cc323b-4437-4c76-adf8-dc2a7dbd4da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" Nov 03 15:47:03.980736 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.980528 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqxrr\" (UniqueName: \"kubernetes.io/projected/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-kube-api-access-sqxrr\") pod \"maas-api-revoke-sub-c405c767-2w5hs\" (UID: \"14cc323b-4437-4c76-adf8-dc2a7dbd4da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" Nov 03 15:47:03.980736 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.980566 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c405c767-2w5hs\" (UID: \"14cc323b-4437-4c76-adf8-dc2a7dbd4da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" Nov 03 15:47:03.981487 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.981438 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c405c767-2w5hs\" (UID: \"14cc323b-4437-4c76-adf8-dc2a7dbd4da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" Nov 03 15:47:03.983759 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:03.983690 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c405c767-2w5hs\" (UID: \"14cc323b-4437-4c76-adf8-dc2a7dbd4da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" Nov 03 15:47:04.042913 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.042885 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqxrr\" (UniqueName: \"kubernetes.io/projected/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-kube-api-access-sqxrr\") pod \"maas-api-revoke-sub-c405c767-2w5hs\" (UID: \"14cc323b-4437-4c76-adf8-dc2a7dbd4da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" Nov 03 15:47:04.128259 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.128184 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" Nov 03 15:47:04.205978 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.205950 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:47:04.286804 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.286766 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-openshift-service-ca\") pod \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " Nov 03 15:47:04.286982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.286817 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-metrics-certs\") pod \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " Nov 03 15:47:04.286982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.286858 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-kube-api-access-kdbq9\" (UniqueName: \"kubernetes.io/projected/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-kube-api-access-kdbq9\") pod \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " Nov 03 15:47:04.286982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.286929 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-maas-api-tls\") pod \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\" (UID: \"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001\") " Nov 03 15:47:04.287262 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.287222 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-openshift-service-ca" pod "fbb5aae0-bbca-4f1f-83c3-875d0ffd9001" (UID: "fbb5aae0-bbca-4f1f-83c3-875d0ffd9001"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:04.288037 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.288009 4290 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-openshift-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:04.289366 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.289333 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-metrics-certs" pod "fbb5aae0-bbca-4f1f-83c3-875d0ffd9001" (UID: "fbb5aae0-bbca-4f1f-83c3-875d0ffd9001"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:47:04.289673 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.289648 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-maas-api-tls" pod "fbb5aae0-bbca-4f1f-83c3-875d0ffd9001" (UID: "fbb5aae0-bbca-4f1f-83c3-875d0ffd9001"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:47:04.290113 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.290087 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-kube-api-access-kdbq9" pod "fbb5aae0-bbca-4f1f-83c3-875d0ffd9001" (UID: "fbb5aae0-bbca-4f1f-83c3-875d0ffd9001"). InnerVolumeSpecName "kube-api-access-kdbq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:04.389692 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.389603 4290 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-metrics-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:04.389692 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.389639 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdbq9\" (UniqueName: \"kubernetes.io/projected/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-kube-api-access-kdbq9\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:04.389692 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.389652 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001-maas-api-tls\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:04.459663 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.459626 4290 generic.go:356] "Generic (PLEG): container finished" podID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" containerID="a15ed5bab533a29f8fc286a0cbe6b43a442ec5ba2e99342e72054484cd97a793" exitCode=0 Nov 03 15:47:04.459869 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.459693 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" event={"ID":"17b58cf9-2cff-4648-86c1-0b1d09acacdd","Type":"ContainerDied","Data":"a15ed5bab533a29f8fc286a0cbe6b43a442ec5ba2e99342e72054484cd97a793"} Nov 03 15:47:04.459869 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.459770 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" event={"ID":"17b58cf9-2cff-4648-86c1-0b1d09acacdd","Type":"ContainerDied","Data":"1c6509ff6cb96b1092590708238aa973232a9a99848e4fb609a5a6be94ce63ed"} Nov 03 15:47:04.459869 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.459768 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n" Nov 03 15:47:04.459869 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.459783 4290 scope.go:122] "RemoveContainer" containerID="a15ed5bab533a29f8fc286a0cbe6b43a442ec5ba2e99342e72054484cd97a793" Nov 03 15:47:04.465107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.462781 4290 generic.go:356] "Generic (PLEG): container finished" podID="fbb5aae0-bbca-4f1f-83c3-875d0ffd9001" containerID="c9f99df6f375629d00e5e0636c703b74935574979c5abbb36e44333b9d112162" exitCode=0 Nov 03 15:47:04.465107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.462889 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" Nov 03 15:47:04.465107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.462919 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" event={"ID":"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001","Type":"ContainerDied","Data":"c9f99df6f375629d00e5e0636c703b74935574979c5abbb36e44333b9d112162"} Nov 03 15:47:04.465107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.462965 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt" event={"ID":"fbb5aae0-bbca-4f1f-83c3-875d0ffd9001","Type":"ContainerDied","Data":"493c53f7a77ba5520bb836c156ffc73df02ef4bbf35932eba22a17e17c98c482"} Nov 03 15:47:04.494288 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.494259 4290 scope.go:122] "RemoveContainer" containerID="a15ed5bab533a29f8fc286a0cbe6b43a442ec5ba2e99342e72054484cd97a793" Nov 03 15:47:04.494661 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:47:04.494633 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a15ed5bab533a29f8fc286a0cbe6b43a442ec5ba2e99342e72054484cd97a793\": container with ID starting with a15ed5bab533a29f8fc286a0cbe6b43a442ec5ba2e99342e72054484cd97a793 not found: ID does not exist" containerID="a15ed5bab533a29f8fc286a0cbe6b43a442ec5ba2e99342e72054484cd97a793" Nov 03 15:47:04.494771 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.494672 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15ed5bab533a29f8fc286a0cbe6b43a442ec5ba2e99342e72054484cd97a793"} err="failed to get container status \"a15ed5bab533a29f8fc286a0cbe6b43a442ec5ba2e99342e72054484cd97a793\": rpc error: code = NotFound desc = could not find container \"a15ed5bab533a29f8fc286a0cbe6b43a442ec5ba2e99342e72054484cd97a793\": container with ID starting with a15ed5bab533a29f8fc286a0cbe6b43a442ec5ba2e99342e72054484cd97a793 not found: ID does not exist" Nov 03 15:47:04.494771 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.494713 4290 scope.go:122] "RemoveContainer" containerID="c9f99df6f375629d00e5e0636c703b74935574979c5abbb36e44333b9d112162" Nov 03 15:47:04.507365 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.507336 4290 scope.go:122] "RemoveContainer" containerID="c9f99df6f375629d00e5e0636c703b74935574979c5abbb36e44333b9d112162" Nov 03 15:47:04.507778 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:47:04.507752 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f99df6f375629d00e5e0636c703b74935574979c5abbb36e44333b9d112162\": container with ID starting with c9f99df6f375629d00e5e0636c703b74935574979c5abbb36e44333b9d112162 not found: ID does not exist" containerID="c9f99df6f375629d00e5e0636c703b74935574979c5abbb36e44333b9d112162" Nov 03 15:47:04.507855 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.507791 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f99df6f375629d00e5e0636c703b74935574979c5abbb36e44333b9d112162"} err="failed to get container status \"c9f99df6f375629d00e5e0636c703b74935574979c5abbb36e44333b9d112162\": rpc error: code = NotFound desc = could not find container \"c9f99df6f375629d00e5e0636c703b74935574979c5abbb36e44333b9d112162\": container with ID starting with c9f99df6f375629d00e5e0636c703b74935574979c5abbb36e44333b9d112162 not found: ID does not exist" Nov 03 15:47:04.570691 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.570654 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs"] Nov 03 15:47:04.571117 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:47:04.571097 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14cc323b_4437_4c76_adf8_dc2a7dbd4da2.slice/crio-1642c595c6f8091a5f89c49624ca29b7dbcace951b979d9af0a0cfd9028026d6 WatchSource:0}: Error finding container 1642c595c6f8091a5f89c49624ca29b7dbcace951b979d9af0a0cfd9028026d6: Status 404 returned error can't find the container with id 1642c595c6f8091a5f89c49624ca29b7dbcace951b979d9af0a0cfd9028026d6 Nov 03 15:47:04.640404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.640323 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n"] Nov 03 15:47:04.656287 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.656253 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/distinct-w4-bc1f07-kserve-8499bfdbb8-rvl5n"] Nov 03 15:47:04.678671 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.678641 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17b58cf9-2cff-4648-86c1-0b1d09acacdd" path="/var/lib/kubelet/pods/17b58cf9-2cff-4648-86c1-0b1d09acacdd/volumes" Nov 03 15:47:04.768984 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.768949 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt"] Nov 03 15:47:04.772765 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:04.772738 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e62ff3-586f46775c-5pfqt"] Nov 03 15:47:05.472165 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:05.472131 4290 generic.go:356] "Generic (PLEG): container finished" podID="14cc323b-4437-4c76-adf8-dc2a7dbd4da2" containerID="9f2633dddf9ec8d07e3abd5dd8cf487b01c041bf892368d6c4ed26e342c6fcdd" exitCode=0 Nov 03 15:47:05.472619 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:05.472295 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" event={"ID":"14cc323b-4437-4c76-adf8-dc2a7dbd4da2","Type":"ContainerDied","Data":"9f2633dddf9ec8d07e3abd5dd8cf487b01c041bf892368d6c4ed26e342c6fcdd"} Nov 03 15:47:05.472619 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:05.472361 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" event={"ID":"14cc323b-4437-4c76-adf8-dc2a7dbd4da2","Type":"ContainerStarted","Data":"1642c595c6f8091a5f89c49624ca29b7dbcace951b979d9af0a0cfd9028026d6"} Nov 03 15:47:06.640799 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:06.640772 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" Nov 03 15:47:06.678032 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:06.677997 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb5aae0-bbca-4f1f-83c3-875d0ffd9001" path="/var/lib/kubelet/pods/fbb5aae0-bbca-4f1f-83c3-875d0ffd9001/volumes" Nov 03 15:47:06.723056 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:06.723020 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-maas-api-openshift-ca\") pod \"14cc323b-4437-4c76-adf8-dc2a7dbd4da2\" (UID: \"14cc323b-4437-4c76-adf8-dc2a7dbd4da2\") " Nov 03 15:47:06.723249 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:06.723066 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-kube-api-access-sqxrr\" (UniqueName: \"kubernetes.io/projected/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-kube-api-access-sqxrr\") pod \"14cc323b-4437-4c76-adf8-dc2a7dbd4da2\" (UID: \"14cc323b-4437-4c76-adf8-dc2a7dbd4da2\") " Nov 03 15:47:06.723314 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:06.723243 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-maas-api-xks-ca\") pod \"14cc323b-4437-4c76-adf8-dc2a7dbd4da2\" (UID: \"14cc323b-4437-4c76-adf8-dc2a7dbd4da2\") " Nov 03 15:47:06.723430 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:06.723401 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-maas-api-openshift-ca" pod "14cc323b-4437-4c76-adf8-dc2a7dbd4da2" (UID: "14cc323b-4437-4c76-adf8-dc2a7dbd4da2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:06.723746 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:06.723718 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:06.725506 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:06.725478 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-kube-api-access-sqxrr" pod "14cc323b-4437-4c76-adf8-dc2a7dbd4da2" (UID: "14cc323b-4437-4c76-adf8-dc2a7dbd4da2"). InnerVolumeSpecName "kube-api-access-sqxrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:06.725598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:06.725495 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-maas-api-xks-ca" pod "14cc323b-4437-4c76-adf8-dc2a7dbd4da2" (UID: "14cc323b-4437-4c76-adf8-dc2a7dbd4da2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:47:06.825016 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:06.824971 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqxrr\" (UniqueName: \"kubernetes.io/projected/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-kube-api-access-sqxrr\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:06.825016 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:06.825008 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14cc323b-4437-4c76-adf8-dc2a7dbd4da2-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.493375 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.493327 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" Nov 03 15:47:07.493587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.493331 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs" event={"ID":"14cc323b-4437-4c76-adf8-dc2a7dbd4da2","Type":"ContainerDied","Data":"1642c595c6f8091a5f89c49624ca29b7dbcace951b979d9af0a0cfd9028026d6"} Nov 03 15:47:07.493587 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.493468 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1642c595c6f8091a5f89c49624ca29b7dbcace951b979d9af0a0cfd9028026d6" Nov 03 15:47:07.495816 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.495777 4290 generic.go:356] "Generic (PLEG): container finished" podID="afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" containerID="321dfb447dbbfe54fa34c09ebb256c6374f30712ab8cc0c712b5cd25dc1334e8" exitCode=0 Nov 03 15:47:07.495972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.495847 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" event={"ID":"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8","Type":"ContainerDied","Data":"321dfb447dbbfe54fa34c09ebb256c6374f30712ab8cc0c712b5cd25dc1334e8"} Nov 03 15:47:07.549212 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.549191 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:47:07.635089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.635059 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-credential-socket\") pod \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " Nov 03 15:47:07.635205 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.635095 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istiod-ca-cert\") pod \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " Nov 03 15:47:07.635205 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.635169 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-podinfo\") pod \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " Nov 03 15:47:07.635205 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.635197 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-workload-certs\") pod \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " Nov 03 15:47:07.635337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.635287 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-kube-api-access-xmbtb\" (UniqueName: \"kubernetes.io/projected/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-kube-api-access-xmbtb\") pod \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " Nov 03 15:47:07.635337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.635317 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-credential-socket" pod "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" (UID: "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:07.635337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.635332 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-envoy\") pod \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " Nov 03 15:47:07.635494 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.635385 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-token\" (UniqueName: \"kubernetes.io/projected/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-token\") pod \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " Nov 03 15:47:07.635494 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.635472 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-workload-socket\") pod \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " Nov 03 15:47:07.635605 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.635523 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-data\") pod \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\" (UID: \"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8\") " Nov 03 15:47:07.635605 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.635568 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-workload-certs" pod "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" (UID: "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:07.635763 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.635618 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istiod-ca-cert" pod "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" (UID: "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:07.635837 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.635816 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-workload-socket" pod "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" (UID: "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:07.636067 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.636045 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-data" pod "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" (UID: "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:07.636231 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.636213 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.636303 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.636240 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.636303 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.636257 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.636303 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.636273 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.636303 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.636288 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.637852 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.637829 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-token" pod "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" (UID: "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:07.637947 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.637903 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-kube-api-access-xmbtb" pod "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" (UID: "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8"). InnerVolumeSpecName "kube-api-access-xmbtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:07.638059 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.638041 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-podinfo" pod "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" (UID: "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:47:07.638101 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.638054 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-envoy" pod "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" (UID: "afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:07.737427 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.737390 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.738017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.737989 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.738120 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.738020 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmbtb\" (UniqueName: \"kubernetes.io/projected/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-kube-api-access-xmbtb\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.738120 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:07.738039 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:08.428267 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:08.428152 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" podUID="ad3f085b-e44a-4f7a-a92e-065c19adf7d8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:47:08.428267 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:08.428167 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" podUID="5ce34d51-acb2-484d-acee-275b15ec6532" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:47:08.508091 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:08.508064 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" Nov 03 15:47:08.508091 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:08.508087 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh" event={"ID":"afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8","Type":"ContainerDied","Data":"36035fd08223b9e46fdeb0e6c4d38a7675e3050a2fe10f3d22ba047dc68650c5"} Nov 03 15:47:08.508349 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:08.508144 4290 scope.go:122] "RemoveContainer" containerID="321dfb447dbbfe54fa34c09ebb256c6374f30712ab8cc0c712b5cd25dc1334e8" Nov 03 15:47:08.704958 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:47:08.704875 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:47:08.726274 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:08.726242 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh"] Nov 03 15:47:08.776418 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:08.776383 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-s6xhh"] Nov 03 15:47:10.596888 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:10.596856 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt"] Nov 03 15:47:10.597377 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:10.597047 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" podUID="d04ac745-e80e-433c-a131-d61612494260" containerName="istio-proxy" containerID="cri-o://13449d28964d81dab54958cb8af42ff6a7f58e2ec5ccdc85839c30cc4ec26602" gracePeriod=30 Nov 03 15:47:10.621807 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:10.621781 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng"] Nov 03 15:47:10.621997 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:10.621977 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" podUID="3f80f0c3-6c46-4763-bd3c-0b4737b40044" containerName="istio-proxy" containerID="cri-o://b1ed34700915c2e076c5d7c900d01a4ba73e87e5bc3ce4c08190d25d75835e42" gracePeriod=30 Nov 03 15:47:10.641132 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:10.641102 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4"] Nov 03 15:47:10.641335 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:10.641312 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" podUID="fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" containerName="istio-proxy" containerID="cri-o://546aa4269313490c85c3b33ffa9e7d1e1f3fccaa77832c4a84221895d641ebbd" gracePeriod=30 Nov 03 15:47:10.727443 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:10.727313 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" path="/var/lib/kubelet/pods/afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8/volumes" Nov 03 15:47:11.451949 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:11.451905 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" podUID="76e830bb-3e77-42b6-9d03-d41e24955b37" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:47:13.335873 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:13.335838 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:47:13.336235 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:13.335905 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" podUID="3f80f0c3-6c46-4763-bd3c-0b4737b40044" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:47:13.390692 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:13.390656 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:47:13.390876 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:13.390750 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" podUID="d04ac745-e80e-433c-a131-d61612494260" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:47:14.691079 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:14.691037 4290 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:47:14.691583 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:14.691116 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" podUID="fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:47:15.130463 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:15.130421 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj"] Nov 03 15:47:15.130745 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:15.130658 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" podUID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" containerName="istio-proxy" containerID="cri-o://79f5c7f5d8852e9341030a17f4b5ca22a9d1c8d56e269fff1dcbfe891ab7a45e" gracePeriod=30 Nov 03 15:47:15.982493 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:15.982438 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:47:16.063523 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.063480 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d04ac745-e80e-433c-a131-d61612494260-istio-token\" (UniqueName: \"kubernetes.io/projected/d04ac745-e80e-433c-a131-d61612494260-istio-token\") pod \"d04ac745-e80e-433c-a131-d61612494260\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " Nov 03 15:47:16.063523 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.063523 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d04ac745-e80e-433c-a131-d61612494260-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04ac745-e80e-433c-a131-d61612494260-istiod-ca-cert\") pod \"d04ac745-e80e-433c-a131-d61612494260\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " Nov 03 15:47:16.063778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.063571 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d04ac745-e80e-433c-a131-d61612494260-kube-api-access-d2fmz\" (UniqueName: \"kubernetes.io/projected/d04ac745-e80e-433c-a131-d61612494260-kube-api-access-d2fmz\") pod \"d04ac745-e80e-433c-a131-d61612494260\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " Nov 03 15:47:16.063778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.063660 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-istio-data\") pod \"d04ac745-e80e-433c-a131-d61612494260\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " Nov 03 15:47:16.063778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.063717 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-workload-certs\") pod \"d04ac745-e80e-433c-a131-d61612494260\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " Nov 03 15:47:16.063951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.063802 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-workload-socket\") pod \"d04ac745-e80e-433c-a131-d61612494260\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " Nov 03 15:47:16.063951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.063882 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d04ac745-e80e-433c-a131-d61612494260-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04ac745-e80e-433c-a131-d61612494260-istio-podinfo\") pod \"d04ac745-e80e-433c-a131-d61612494260\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " Nov 03 15:47:16.064067 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.063990 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-credential-socket\") pod \"d04ac745-e80e-433c-a131-d61612494260\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " Nov 03 15:47:16.064067 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.064036 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-istio-envoy\") pod \"d04ac745-e80e-433c-a131-d61612494260\" (UID: \"d04ac745-e80e-433c-a131-d61612494260\") " Nov 03 15:47:16.064190 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.064143 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-workload-certs" pod "d04ac745-e80e-433c-a131-d61612494260" (UID: "d04ac745-e80e-433c-a131-d61612494260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.064247 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.064204 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-workload-socket" pod "d04ac745-e80e-433c-a131-d61612494260" (UID: "d04ac745-e80e-433c-a131-d61612494260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.064542 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.064519 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.064668 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.064544 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.064668 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.064593 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04ac745-e80e-433c-a131-d61612494260-istiod-ca-cert" pod "d04ac745-e80e-433c-a131-d61612494260" (UID: "d04ac745-e80e-433c-a131-d61612494260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:16.065027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.064993 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-istio-data" pod "d04ac745-e80e-433c-a131-d61612494260" (UID: "d04ac745-e80e-433c-a131-d61612494260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.065027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.065006 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-credential-socket" pod "d04ac745-e80e-433c-a131-d61612494260" (UID: "d04ac745-e80e-433c-a131-d61612494260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.069011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.067438 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04ac745-e80e-433c-a131-d61612494260-kube-api-access-d2fmz" pod "d04ac745-e80e-433c-a131-d61612494260" (UID: "d04ac745-e80e-433c-a131-d61612494260"). InnerVolumeSpecName "kube-api-access-d2fmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:16.069011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.067443 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-istio-envoy" pod "d04ac745-e80e-433c-a131-d61612494260" (UID: "d04ac745-e80e-433c-a131-d61612494260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.069011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.068301 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04ac745-e80e-433c-a131-d61612494260-istio-token" pod "d04ac745-e80e-433c-a131-d61612494260" (UID: "d04ac745-e80e-433c-a131-d61612494260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:16.069776 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.069738 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d04ac745-e80e-433c-a131-d61612494260-istio-podinfo" pod "d04ac745-e80e-433c-a131-d61612494260" (UID: "d04ac745-e80e-433c-a131-d61612494260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:47:16.079541 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.079262 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:47:16.081373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.081292 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" Nov 03 15:47:16.100219 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.100185 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:47:16.165298 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.165265 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-data\") pod \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " Nov 03 15:47:16.165503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.165346 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-podinfo\") pod \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " Nov 03 15:47:16.165503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.165434 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-token\" (UniqueName: \"kubernetes.io/projected/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-token\") pod \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " Nov 03 15:47:16.165503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.165475 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-credential-socket\") pod \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " Nov 03 15:47:16.165678 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.165514 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istiod-ca-cert\") pod \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " Nov 03 15:47:16.165678 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.165572 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-credential-socket\") pod \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " Nov 03 15:47:16.165678 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.165605 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-workload-certs\") pod \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " Nov 03 15:47:16.165678 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.165647 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-envoy\") pod \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " Nov 03 15:47:16.165934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.165676 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e830bb-3e77-42b6-9d03-d41e24955b37-kube-api-access-jn5dq\" (UniqueName: \"kubernetes.io/projected/76e830bb-3e77-42b6-9d03-d41e24955b37-kube-api-access-jn5dq\") pod \"76e830bb-3e77-42b6-9d03-d41e24955b37\" (UID: \"76e830bb-3e77-42b6-9d03-d41e24955b37\") " Nov 03 15:47:16.165934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.165742 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76e830bb-3e77-42b6-9d03-d41e24955b37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e830bb-3e77-42b6-9d03-d41e24955b37-plugins-config-volume\") pod \"76e830bb-3e77-42b6-9d03-d41e24955b37\" (UID: \"76e830bb-3e77-42b6-9d03-d41e24955b37\") " Nov 03 15:47:16.165934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.165775 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f80f0c3-6c46-4763-bd3c-0b4737b40044-kube-api-access-dhvd2\" (UniqueName: \"kubernetes.io/projected/3f80f0c3-6c46-4763-bd3c-0b4737b40044-kube-api-access-dhvd2\") pod \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " Nov 03 15:47:16.165934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.165852 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istiod-ca-cert\") pod \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " Nov 03 15:47:16.165934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.165884 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-workload-socket\") pod \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " Nov 03 15:47:16.165934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.165928 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-envoy\") pod \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " Nov 03 15:47:16.166241 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.165976 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-workload-certs\") pod \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " Nov 03 15:47:16.166241 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.166007 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-token\" (UniqueName: \"kubernetes.io/projected/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-token\") pod \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\" (UID: \"3f80f0c3-6c46-4763-bd3c-0b4737b40044\") " Nov 03 15:47:16.166241 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.166048 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-podinfo\") pod \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " Nov 03 15:47:16.166241 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.166105 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-data\") pod \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " Nov 03 15:47:16.166241 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.166160 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-workload-socket\") pod \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " Nov 03 15:47:16.166241 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.166225 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-kube-api-access-94vzv\" (UniqueName: \"kubernetes.io/projected/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-kube-api-access-94vzv\") pod \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\" (UID: \"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c\") " Nov 03 15:47:16.166607 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.166575 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istiod-ca-cert" pod "3f80f0c3-6c46-4763-bd3c-0b4737b40044" (UID: "3f80f0c3-6c46-4763-bd3c-0b4737b40044"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:16.166853 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.166827 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04ac745-e80e-433c-a131-d61612494260-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.166957 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.166858 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04ac745-e80e-433c-a131-d61612494260-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.166957 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.166875 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2fmz\" (UniqueName: \"kubernetes.io/projected/d04ac745-e80e-433c-a131-d61612494260-kube-api-access-d2fmz\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.166957 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.166891 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.166957 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.166906 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.166957 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.166920 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04ac745-e80e-433c-a131-d61612494260-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.166957 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.166934 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.167170 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.167039 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-data" pod "3f80f0c3-6c46-4763-bd3c-0b4737b40044" (UID: "3f80f0c3-6c46-4763-bd3c-0b4737b40044"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.167219 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.167190 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04ac745-e80e-433c-a131-d61612494260-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.167273 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.167244 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-workload-certs" pod "3f80f0c3-6c46-4763-bd3c-0b4737b40044" (UID: "3f80f0c3-6c46-4763-bd3c-0b4737b40044"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.168203 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.168173 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e830bb-3e77-42b6-9d03-d41e24955b37-plugins-config-volume" pod "76e830bb-3e77-42b6-9d03-d41e24955b37" (UID: "76e830bb-3e77-42b6-9d03-d41e24955b37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:16.170253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.170220 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-credential-socket" pod "3f80f0c3-6c46-4763-bd3c-0b4737b40044" (UID: "3f80f0c3-6c46-4763-bd3c-0b4737b40044"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.170446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.170423 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-credential-socket" pod "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" (UID: "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.170748 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.170725 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istiod-ca-cert" pod "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" (UID: "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:16.170999 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.170975 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-data" pod "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" (UID: "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.174277 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.172628 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-envoy" pod "3f80f0c3-6c46-4763-bd3c-0b4737b40044" (UID: "3f80f0c3-6c46-4763-bd3c-0b4737b40044"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.174277 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.172880 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-workload-certs" pod "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" (UID: "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.174277 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.173053 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-workload-socket" pod "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" (UID: "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.174277 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.173227 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-workload-socket" pod "3f80f0c3-6c46-4763-bd3c-0b4737b40044" (UID: "3f80f0c3-6c46-4763-bd3c-0b4737b40044"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.174277 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.173608 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-kube-api-access-94vzv" pod "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" (UID: "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c"). InnerVolumeSpecName "kube-api-access-94vzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:16.175131 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.175102 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-podinfo" pod "3f80f0c3-6c46-4763-bd3c-0b4737b40044" (UID: "3f80f0c3-6c46-4763-bd3c-0b4737b40044"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:47:16.175294 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.175258 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f80f0c3-6c46-4763-bd3c-0b4737b40044-kube-api-access-dhvd2" pod "3f80f0c3-6c46-4763-bd3c-0b4737b40044" (UID: "3f80f0c3-6c46-4763-bd3c-0b4737b40044"). InnerVolumeSpecName "kube-api-access-dhvd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:16.176076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.176040 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-token" pod "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" (UID: "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:16.176569 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.176544 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-token" pod "3f80f0c3-6c46-4763-bd3c-0b4737b40044" (UID: "3f80f0c3-6c46-4763-bd3c-0b4737b40044"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:16.176666 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.176570 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-podinfo" pod "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" (UID: "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:47:16.176857 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.176836 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-envoy" pod "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" (UID: "fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.177004 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.176984 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e830bb-3e77-42b6-9d03-d41e24955b37-kube-api-access-jn5dq" pod "76e830bb-3e77-42b6-9d03-d41e24955b37" (UID: "76e830bb-3e77-42b6-9d03-d41e24955b37"). InnerVolumeSpecName "kube-api-access-jn5dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:16.268502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268409 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268452 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268466 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268479 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268491 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268504 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268518 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268534 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn5dq\" (UniqueName: \"kubernetes.io/projected/76e830bb-3e77-42b6-9d03-d41e24955b37-kube-api-access-jn5dq\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268547 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e830bb-3e77-42b6-9d03-d41e24955b37-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268561 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhvd2\" (UniqueName: \"kubernetes.io/projected/3f80f0c3-6c46-4763-bd3c-0b4737b40044-kube-api-access-dhvd2\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268575 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268590 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268604 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268616 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268630 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268643 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268657 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268670 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94vzv\" (UniqueName: \"kubernetes.io/projected/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c-kube-api-access-94vzv\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.268893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.268682 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f80f0c3-6c46-4763-bd3c-0b4737b40044-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.586929 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.586841 4290 generic.go:356] "Generic (PLEG): container finished" podID="3f80f0c3-6c46-4763-bd3c-0b4737b40044" containerID="b1ed34700915c2e076c5d7c900d01a4ba73e87e5bc3ce4c08190d25d75835e42" exitCode=0 Nov 03 15:47:16.586929 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.586905 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" Nov 03 15:47:16.587166 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.586926 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" event={"ID":"3f80f0c3-6c46-4763-bd3c-0b4737b40044","Type":"ContainerDied","Data":"b1ed34700915c2e076c5d7c900d01a4ba73e87e5bc3ce4c08190d25d75835e42"} Nov 03 15:47:16.587166 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.587020 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng" event={"ID":"3f80f0c3-6c46-4763-bd3c-0b4737b40044","Type":"ContainerDied","Data":"0958850f6b2c4c27e0a9343221c4f67fe16b58829a702f53f7f16cff5c605fe8"} Nov 03 15:47:16.587166 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.587045 4290 scope.go:122] "RemoveContainer" containerID="b1ed34700915c2e076c5d7c900d01a4ba73e87e5bc3ce4c08190d25d75835e42" Nov 03 15:47:16.588879 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.588812 4290 generic.go:356] "Generic (PLEG): container finished" podID="fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" containerID="546aa4269313490c85c3b33ffa9e7d1e1f3fccaa77832c4a84221895d641ebbd" exitCode=0 Nov 03 15:47:16.588879 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.588873 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" Nov 03 15:47:16.589047 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.588879 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" event={"ID":"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c","Type":"ContainerDied","Data":"546aa4269313490c85c3b33ffa9e7d1e1f3fccaa77832c4a84221895d641ebbd"} Nov 03 15:47:16.589047 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.588933 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4" event={"ID":"fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c","Type":"ContainerDied","Data":"6fdb33126da2cf5abea89a223f813e7463650e0c90195017e4ee0cb136b95060"} Nov 03 15:47:16.590562 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.590535 4290 generic.go:356] "Generic (PLEG): container finished" podID="76e830bb-3e77-42b6-9d03-d41e24955b37" containerID="a9575c81007505760bcb470892761d230d58ebe5621e4581bfb881ac6312c156" exitCode=0 Nov 03 15:47:16.590662 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.590597 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" Nov 03 15:47:16.590662 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.590631 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" event={"ID":"76e830bb-3e77-42b6-9d03-d41e24955b37","Type":"ContainerDied","Data":"a9575c81007505760bcb470892761d230d58ebe5621e4581bfb881ac6312c156"} Nov 03 15:47:16.590823 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.590670 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt" event={"ID":"76e830bb-3e77-42b6-9d03-d41e24955b37","Type":"ContainerDied","Data":"26ae5e9dc94856f19f5962cc971b268b375c2f34e7526cc1c36d6fc4a908f59f"} Nov 03 15:47:16.592206 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.592183 4290 generic.go:356] "Generic (PLEG): container finished" podID="d04ac745-e80e-433c-a131-d61612494260" containerID="13449d28964d81dab54958cb8af42ff6a7f58e2ec5ccdc85839c30cc4ec26602" exitCode=0 Nov 03 15:47:16.592319 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.592234 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" Nov 03 15:47:16.592319 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.592301 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" event={"ID":"d04ac745-e80e-433c-a131-d61612494260","Type":"ContainerDied","Data":"13449d28964d81dab54958cb8af42ff6a7f58e2ec5ccdc85839c30cc4ec26602"} Nov 03 15:47:16.592437 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.592335 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt" event={"ID":"d04ac745-e80e-433c-a131-d61612494260","Type":"ContainerDied","Data":"d0ddf178a1c9b4647c6c47efe94d0e9b09a195db51309f4e6dd1d0c6d2918559"} Nov 03 15:47:16.601399 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.601317 4290 scope.go:122] "RemoveContainer" containerID="b1ed34700915c2e076c5d7c900d01a4ba73e87e5bc3ce4c08190d25d75835e42" Nov 03 15:47:16.601792 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:47:16.601690 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ed34700915c2e076c5d7c900d01a4ba73e87e5bc3ce4c08190d25d75835e42\": container with ID starting with b1ed34700915c2e076c5d7c900d01a4ba73e87e5bc3ce4c08190d25d75835e42 not found: ID does not exist" containerID="b1ed34700915c2e076c5d7c900d01a4ba73e87e5bc3ce4c08190d25d75835e42" Nov 03 15:47:16.601792 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.601756 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ed34700915c2e076c5d7c900d01a4ba73e87e5bc3ce4c08190d25d75835e42"} err="failed to get container status \"b1ed34700915c2e076c5d7c900d01a4ba73e87e5bc3ce4c08190d25d75835e42\": rpc error: code = NotFound desc = could not find container \"b1ed34700915c2e076c5d7c900d01a4ba73e87e5bc3ce4c08190d25d75835e42\": container with ID starting with b1ed34700915c2e076c5d7c900d01a4ba73e87e5bc3ce4c08190d25d75835e42 not found: ID does not exist" Nov 03 15:47:16.601792 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.601779 4290 scope.go:122] "RemoveContainer" containerID="546aa4269313490c85c3b33ffa9e7d1e1f3fccaa77832c4a84221895d641ebbd" Nov 03 15:47:16.614989 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.614961 4290 scope.go:122] "RemoveContainer" containerID="546aa4269313490c85c3b33ffa9e7d1e1f3fccaa77832c4a84221895d641ebbd" Nov 03 15:47:16.615328 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:47:16.615291 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546aa4269313490c85c3b33ffa9e7d1e1f3fccaa77832c4a84221895d641ebbd\": container with ID starting with 546aa4269313490c85c3b33ffa9e7d1e1f3fccaa77832c4a84221895d641ebbd not found: ID does not exist" containerID="546aa4269313490c85c3b33ffa9e7d1e1f3fccaa77832c4a84221895d641ebbd" Nov 03 15:47:16.615404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.615320 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546aa4269313490c85c3b33ffa9e7d1e1f3fccaa77832c4a84221895d641ebbd"} err="failed to get container status \"546aa4269313490c85c3b33ffa9e7d1e1f3fccaa77832c4a84221895d641ebbd\": rpc error: code = NotFound desc = could not find container \"546aa4269313490c85c3b33ffa9e7d1e1f3fccaa77832c4a84221895d641ebbd\": container with ID starting with 546aa4269313490c85c3b33ffa9e7d1e1f3fccaa77832c4a84221895d641ebbd not found: ID does not exist" Nov 03 15:47:16.615404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.615343 4290 scope.go:122] "RemoveContainer" containerID="a9575c81007505760bcb470892761d230d58ebe5621e4581bfb881ac6312c156" Nov 03 15:47:16.646027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.646003 4290 scope.go:122] "RemoveContainer" containerID="a9575c81007505760bcb470892761d230d58ebe5621e4581bfb881ac6312c156" Nov 03 15:47:16.646377 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:47:16.646350 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9575c81007505760bcb470892761d230d58ebe5621e4581bfb881ac6312c156\": container with ID starting with a9575c81007505760bcb470892761d230d58ebe5621e4581bfb881ac6312c156 not found: ID does not exist" containerID="a9575c81007505760bcb470892761d230d58ebe5621e4581bfb881ac6312c156" Nov 03 15:47:16.646508 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.646382 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9575c81007505760bcb470892761d230d58ebe5621e4581bfb881ac6312c156"} err="failed to get container status \"a9575c81007505760bcb470892761d230d58ebe5621e4581bfb881ac6312c156\": rpc error: code = NotFound desc = could not find container \"a9575c81007505760bcb470892761d230d58ebe5621e4581bfb881ac6312c156\": container with ID starting with a9575c81007505760bcb470892761d230d58ebe5621e4581bfb881ac6312c156 not found: ID does not exist" Nov 03 15:47:16.646508 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.646401 4290 scope.go:122] "RemoveContainer" containerID="13449d28964d81dab54958cb8af42ff6a7f58e2ec5ccdc85839c30cc4ec26602" Nov 03 15:47:16.657076 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.657054 4290 scope.go:122] "RemoveContainer" containerID="13449d28964d81dab54958cb8af42ff6a7f58e2ec5ccdc85839c30cc4ec26602" Nov 03 15:47:16.657414 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:47:16.657395 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13449d28964d81dab54958cb8af42ff6a7f58e2ec5ccdc85839c30cc4ec26602\": container with ID starting with 13449d28964d81dab54958cb8af42ff6a7f58e2ec5ccdc85839c30cc4ec26602 not found: ID does not exist" containerID="13449d28964d81dab54958cb8af42ff6a7f58e2ec5ccdc85839c30cc4ec26602" Nov 03 15:47:16.657491 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.657421 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13449d28964d81dab54958cb8af42ff6a7f58e2ec5ccdc85839c30cc4ec26602"} err="failed to get container status \"13449d28964d81dab54958cb8af42ff6a7f58e2ec5ccdc85839c30cc4ec26602\": rpc error: code = NotFound desc = could not find container \"13449d28964d81dab54958cb8af42ff6a7f58e2ec5ccdc85839c30cc4ec26602\": container with ID starting with 13449d28964d81dab54958cb8af42ff6a7f58e2ec5ccdc85839c30cc4ec26602 not found: ID does not exist" Nov 03 15:47:16.757187 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.757156 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt"] Nov 03 15:47:16.763670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.763642 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-vmngt"] Nov 03 15:47:16.888136 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.888098 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng"] Nov 03 15:47:16.897422 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:16.897391 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-kwmng"] Nov 03 15:47:17.007961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:17.007923 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4"] Nov 03 15:47:17.016558 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:17.016528 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-xmlq4"] Nov 03 15:47:17.136292 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:17.136259 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt"] Nov 03 15:47:17.143040 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:17.142973 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e62ff3-68789f7667-75wxt"] Nov 03 15:47:17.555103 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:17.555015 4290 scope.go:122] "RemoveContainer" containerID="67d567777658282fdb3a7922197bae35ad2eb48cc0b288a9cf67fc8f2a3b787d" Nov 03 15:47:18.428770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:18.428735 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" podUID="5ce34d51-acb2-484d-acee-275b15ec6532" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:47:18.428770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:18.428769 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" podUID="ad3f085b-e44a-4f7a-a92e-065c19adf7d8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:47:18.678205 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:18.678170 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f80f0c3-6c46-4763-bd3c-0b4737b40044" path="/var/lib/kubelet/pods/3f80f0c3-6c46-4763-bd3c-0b4737b40044/volumes" Nov 03 15:47:18.678625 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:18.678610 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e830bb-3e77-42b6-9d03-d41e24955b37" path="/var/lib/kubelet/pods/76e830bb-3e77-42b6-9d03-d41e24955b37/volumes" Nov 03 15:47:18.679083 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:18.679044 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d04ac745-e80e-433c-a131-d61612494260" path="/var/lib/kubelet/pods/d04ac745-e80e-433c-a131-d61612494260/volumes" Nov 03 15:47:18.679811 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:18.679795 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" path="/var/lib/kubelet/pods/fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c/volumes" Nov 03 15:47:20.597746 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.597695 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:47:20.644871 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.643793 4290 generic.go:356] "Generic (PLEG): container finished" podID="5ce34d51-acb2-484d-acee-275b15ec6532" containerID="171fd0f269122493c3e40d6c80e6d504bc8d19d4f5439d998e5bfa112cf75c37" exitCode=0 Nov 03 15:47:20.644871 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.643953 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" event={"ID":"5ce34d51-acb2-484d-acee-275b15ec6532","Type":"ContainerDied","Data":"171fd0f269122493c3e40d6c80e6d504bc8d19d4f5439d998e5bfa112cf75c37"} Nov 03 15:47:20.646119 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.646092 4290 generic.go:356] "Generic (PLEG): container finished" podID="ad3f085b-e44a-4f7a-a92e-065c19adf7d8" containerID="2eed4a4835c16f79ef0d8917c228f7939821e72dd69278a6a33230da3d11288e" exitCode=0 Nov 03 15:47:20.646253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.646167 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" event={"ID":"ad3f085b-e44a-4f7a-a92e-065c19adf7d8","Type":"ContainerDied","Data":"2eed4a4835c16f79ef0d8917c228f7939821e72dd69278a6a33230da3d11288e"} Nov 03 15:47:20.647895 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.647881 4290 generic.go:356] "Generic (PLEG): container finished" podID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" containerID="79f5c7f5d8852e9341030a17f4b5ca22a9d1c8d56e269fff1dcbfe891ab7a45e" exitCode=0 Nov 03 15:47:20.647966 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.647952 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" event={"ID":"b4e58dd3-dccf-45da-93fd-e4bbabf337db","Type":"ContainerDied","Data":"79f5c7f5d8852e9341030a17f4b5ca22a9d1c8d56e269fff1dcbfe891ab7a45e"} Nov 03 15:47:20.648020 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.647980 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" Nov 03 15:47:20.648020 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.647989 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj" event={"ID":"b4e58dd3-dccf-45da-93fd-e4bbabf337db","Type":"ContainerDied","Data":"55d6bd77ad20a3a06c33921b7c3c2dc5d05ed8084606d24f69deb84dffd49da2"} Nov 03 15:47:20.648125 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.648011 4290 scope.go:122] "RemoveContainer" containerID="79f5c7f5d8852e9341030a17f4b5ca22a9d1c8d56e269fff1dcbfe891ab7a45e" Nov 03 15:47:20.657131 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.657112 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" Nov 03 15:47:20.661027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.661012 4290 scope.go:122] "RemoveContainer" containerID="79f5c7f5d8852e9341030a17f4b5ca22a9d1c8d56e269fff1dcbfe891ab7a45e" Nov 03 15:47:20.661346 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:47:20.661319 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f5c7f5d8852e9341030a17f4b5ca22a9d1c8d56e269fff1dcbfe891ab7a45e\": container with ID starting with 79f5c7f5d8852e9341030a17f4b5ca22a9d1c8d56e269fff1dcbfe891ab7a45e not found: ID does not exist" containerID="79f5c7f5d8852e9341030a17f4b5ca22a9d1c8d56e269fff1dcbfe891ab7a45e" Nov 03 15:47:20.661395 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.661356 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f5c7f5d8852e9341030a17f4b5ca22a9d1c8d56e269fff1dcbfe891ab7a45e"} err="failed to get container status \"79f5c7f5d8852e9341030a17f4b5ca22a9d1c8d56e269fff1dcbfe891ab7a45e\": rpc error: code = NotFound desc = could not find container \"79f5c7f5d8852e9341030a17f4b5ca22a9d1c8d56e269fff1dcbfe891ab7a45e\": container with ID starting with 79f5c7f5d8852e9341030a17f4b5ca22a9d1c8d56e269fff1dcbfe891ab7a45e not found: ID does not exist" Nov 03 15:47:20.752011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.751985 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" Nov 03 15:47:20.785963 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.785929 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3f085b-e44a-4f7a-a92e-065c19adf7d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3f085b-e44a-4f7a-a92e-065c19adf7d8-plugins-config-volume\") pod \"ad3f085b-e44a-4f7a-a92e-065c19adf7d8\" (UID: \"ad3f085b-e44a-4f7a-a92e-065c19adf7d8\") " Nov 03 15:47:20.786147 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.786035 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e58dd3-dccf-45da-93fd-e4bbabf337db-kube-api-access-vscxf\" (UniqueName: \"kubernetes.io/projected/b4e58dd3-dccf-45da-93fd-e4bbabf337db-kube-api-access-vscxf\") pod \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " Nov 03 15:47:20.786147 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.786068 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-token\" (UniqueName: \"kubernetes.io/projected/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-token\") pod \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " Nov 03 15:47:20.786147 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.786100 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-data\") pod \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " Nov 03 15:47:20.786147 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.786133 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-workload-certs\") pod \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " Nov 03 15:47:20.786373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.786195 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-workload-socket\") pod \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " Nov 03 15:47:20.786373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.786296 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3f085b-e44a-4f7a-a92e-065c19adf7d8-kube-api-access-qvjrw\" (UniqueName: \"kubernetes.io/projected/ad3f085b-e44a-4f7a-a92e-065c19adf7d8-kube-api-access-qvjrw\") pod \"ad3f085b-e44a-4f7a-a92e-065c19adf7d8\" (UID: \"ad3f085b-e44a-4f7a-a92e-065c19adf7d8\") " Nov 03 15:47:20.786373 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.786301 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3f085b-e44a-4f7a-a92e-065c19adf7d8-plugins-config-volume" pod "ad3f085b-e44a-4f7a-a92e-065c19adf7d8" (UID: "ad3f085b-e44a-4f7a-a92e-065c19adf7d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:20.786525 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.786385 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-credential-socket\") pod \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " Nov 03 15:47:20.786525 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.786415 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istiod-ca-cert\") pod \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " Nov 03 15:47:20.786525 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.786450 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-podinfo\") pod \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " Nov 03 15:47:20.786525 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.786480 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-envoy\") pod \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\" (UID: \"b4e58dd3-dccf-45da-93fd-e4bbabf337db\") " Nov 03 15:47:20.786525 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.786510 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-data" pod "b4e58dd3-dccf-45da-93fd-e4bbabf337db" (UID: "b4e58dd3-dccf-45da-93fd-e4bbabf337db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:20.786796 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.786619 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-workload-certs" pod "b4e58dd3-dccf-45da-93fd-e4bbabf337db" (UID: "b4e58dd3-dccf-45da-93fd-e4bbabf337db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:20.786796 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.786723 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-credential-socket" pod "b4e58dd3-dccf-45da-93fd-e4bbabf337db" (UID: "b4e58dd3-dccf-45da-93fd-e4bbabf337db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:20.786910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.786896 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istiod-ca-cert" pod "b4e58dd3-dccf-45da-93fd-e4bbabf337db" (UID: "b4e58dd3-dccf-45da-93fd-e4bbabf337db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:20.787107 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.787069 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-workload-socket" pod "b4e58dd3-dccf-45da-93fd-e4bbabf337db" (UID: "b4e58dd3-dccf-45da-93fd-e4bbabf337db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:20.787191 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.787120 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.787191 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.787142 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.787191 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.787158 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3f085b-e44a-4f7a-a92e-065c19adf7d8-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.787191 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.787172 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.787191 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.787181 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.789351 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.789325 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-token" pod "b4e58dd3-dccf-45da-93fd-e4bbabf337db" (UID: "b4e58dd3-dccf-45da-93fd-e4bbabf337db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:20.789611 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.789580 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-envoy" pod "b4e58dd3-dccf-45da-93fd-e4bbabf337db" (UID: "b4e58dd3-dccf-45da-93fd-e4bbabf337db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:20.789761 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.789666 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-podinfo" pod "b4e58dd3-dccf-45da-93fd-e4bbabf337db" (UID: "b4e58dd3-dccf-45da-93fd-e4bbabf337db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:47:20.789821 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.789800 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e58dd3-dccf-45da-93fd-e4bbabf337db-kube-api-access-vscxf" pod "b4e58dd3-dccf-45da-93fd-e4bbabf337db" (UID: "b4e58dd3-dccf-45da-93fd-e4bbabf337db"). InnerVolumeSpecName "kube-api-access-vscxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:20.789965 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.789939 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3f085b-e44a-4f7a-a92e-065c19adf7d8-kube-api-access-qvjrw" pod "ad3f085b-e44a-4f7a-a92e-065c19adf7d8" (UID: "ad3f085b-e44a-4f7a-a92e-065c19adf7d8"). InnerVolumeSpecName "kube-api-access-qvjrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:20.888118 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.888084 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce34d51-acb2-484d-acee-275b15ec6532-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce34d51-acb2-484d-acee-275b15ec6532-plugins-config-volume\") pod \"5ce34d51-acb2-484d-acee-275b15ec6532\" (UID: \"5ce34d51-acb2-484d-acee-275b15ec6532\") " Nov 03 15:47:20.888303 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.888271 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce34d51-acb2-484d-acee-275b15ec6532-kube-api-access-twmwh\" (UniqueName: \"kubernetes.io/projected/5ce34d51-acb2-484d-acee-275b15ec6532-kube-api-access-twmwh\") pod \"5ce34d51-acb2-484d-acee-275b15ec6532\" (UID: \"5ce34d51-acb2-484d-acee-275b15ec6532\") " Nov 03 15:47:20.888784 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.888750 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vscxf\" (UniqueName: \"kubernetes.io/projected/b4e58dd3-dccf-45da-93fd-e4bbabf337db-kube-api-access-vscxf\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.888784 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.888781 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.888961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.888786 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce34d51-acb2-484d-acee-275b15ec6532-plugins-config-volume" pod "5ce34d51-acb2-484d-acee-275b15ec6532" (UID: "5ce34d51-acb2-484d-acee-275b15ec6532"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:20.888961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.888797 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.888961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.888811 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvjrw\" (UniqueName: \"kubernetes.io/projected/ad3f085b-e44a-4f7a-a92e-065c19adf7d8-kube-api-access-qvjrw\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.888961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.888825 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.888961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.888835 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e58dd3-dccf-45da-93fd-e4bbabf337db-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.891343 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.891303 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce34d51-acb2-484d-acee-275b15ec6532-kube-api-access-twmwh" pod "5ce34d51-acb2-484d-acee-275b15ec6532" (UID: "5ce34d51-acb2-484d-acee-275b15ec6532"). InnerVolumeSpecName "kube-api-access-twmwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:20.990173 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.990143 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce34d51-acb2-484d-acee-275b15ec6532-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.990173 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:20.990169 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twmwh\" (UniqueName: \"kubernetes.io/projected/5ce34d51-acb2-484d-acee-275b15ec6532-kube-api-access-twmwh\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:21.181957 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:21.181928 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj"] Nov 03 15:47:21.185244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:21.185222 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-kz5bj"] Nov 03 15:47:21.469747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:21.469646 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj"] Nov 03 15:47:21.469969 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:21.469892 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" containerID="cri-o://48472a61f95b896b487d15500b7ef9923da33dee9121d930c06e01579a83fbe9" gracePeriod=30 Nov 03 15:47:21.656750 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:21.656691 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" Nov 03 15:47:21.656750 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:21.656738 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t" event={"ID":"ad3f085b-e44a-4f7a-a92e-065c19adf7d8","Type":"ContainerDied","Data":"4d3cd855d557b38490a53e8f164d008f5912970f42fd53e1e77aaec83645e86f"} Nov 03 15:47:21.657301 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:21.656811 4290 scope.go:122] "RemoveContainer" containerID="2eed4a4835c16f79ef0d8917c228f7939821e72dd69278a6a33230da3d11288e" Nov 03 15:47:21.659745 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:21.659693 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" Nov 03 15:47:21.659858 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:21.659750 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft" event={"ID":"5ce34d51-acb2-484d-acee-275b15ec6532","Type":"ContainerDied","Data":"be917eae63db53cdec1c556bf0d5a4e1b316bceb88c790e613cc3b1009e4d6d1"} Nov 03 15:47:21.667935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:21.667915 4290 scope.go:122] "RemoveContainer" containerID="171fd0f269122493c3e40d6c80e6d504bc8d19d4f5439d998e5bfa112cf75c37" Nov 03 15:47:21.824157 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:21.824081 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t"] Nov 03 15:47:21.829672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:21.829649 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bc1f07-7fdbf965df-xg47t"] Nov 03 15:47:21.941762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:21.941670 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft"] Nov 03 15:47:21.945278 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:21.945249 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bc1f07-78b987d5f9-bslft"] Nov 03 15:47:22.683538 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:22.683501 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce34d51-acb2-484d-acee-275b15ec6532" path="/var/lib/kubelet/pods/5ce34d51-acb2-484d-acee-275b15ec6532/volumes" Nov 03 15:47:22.683961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:22.683896 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3f085b-e44a-4f7a-a92e-065c19adf7d8" path="/var/lib/kubelet/pods/ad3f085b-e44a-4f7a-a92e-065c19adf7d8/volumes" Nov 03 15:47:22.684301 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:22.684286 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" path="/var/lib/kubelet/pods/b4e58dd3-dccf-45da-93fd-e4bbabf337db/volumes" Nov 03 15:47:26.704598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.704554 4290 generic.go:356] "Generic (PLEG): container finished" podID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerID="48472a61f95b896b487d15500b7ef9923da33dee9121d930c06e01579a83fbe9" exitCode=0 Nov 03 15:47:26.705104 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.704659 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" event={"ID":"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8","Type":"ContainerDied","Data":"48472a61f95b896b487d15500b7ef9923da33dee9121d930c06e01579a83fbe9"} Nov 03 15:47:26.760285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.760224 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:47:26.849783 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.849738 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-data\") pod \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " Nov 03 15:47:26.849998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.849802 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-kube-api-access-chwm5\" (UniqueName: \"kubernetes.io/projected/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-kube-api-access-chwm5\") pod \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " Nov 03 15:47:26.849998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.849833 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-workload-certs\") pod \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " Nov 03 15:47:26.849998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.849855 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istiod-ca-cert\") pod \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " Nov 03 15:47:26.849998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.849891 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-token\" (UniqueName: \"kubernetes.io/projected/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-token\") pod \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " Nov 03 15:47:26.849998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.849920 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-workload-socket\") pod \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " Nov 03 15:47:26.850285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.850046 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-credential-socket\") pod \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " Nov 03 15:47:26.850285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.850063 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-data" pod "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" (UID: "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:26.850285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.850097 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-podinfo\") pod \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " Nov 03 15:47:26.850285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.850206 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-envoy\") pod \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\" (UID: \"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8\") " Nov 03 15:47:26.850285 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.850233 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-workload-certs" pod "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" (UID: "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:26.850515 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.850398 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-workload-socket" pod "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" (UID: "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:26.850515 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.850402 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istiod-ca-cert" pod "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" (UID: "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:26.850633 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.850395 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-credential-socket" pod "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" (UID: "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:26.850826 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.850805 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:26.850826 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.850826 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:26.851033 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.850840 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:26.851033 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.850854 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:26.851033 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.850866 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:26.852983 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.852955 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-token" pod "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" (UID: "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:26.853910 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.853886 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-kube-api-access-chwm5" pod "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" (UID: "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8"). InnerVolumeSpecName "kube-api-access-chwm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:26.854002 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.853935 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-podinfo" pod "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" (UID: "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:47:26.854078 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.854060 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-envoy" pod "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" (UID: "ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:26.952243 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.952211 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:26.952243 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.952241 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:26.952444 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.952255 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:26.952444 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:26.952268 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chwm5\" (UniqueName: \"kubernetes.io/projected/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8-kube-api-access-chwm5\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:27.718381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:27.718347 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" Nov 03 15:47:27.718915 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:27.718358 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj" event={"ID":"ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8","Type":"ContainerDied","Data":"28df500608325cf572957a867791b001f28c01ca7dc47bb949d8d5c33151634c"} Nov 03 15:47:27.718915 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:27.718496 4290 scope.go:122] "RemoveContainer" containerID="48472a61f95b896b487d15500b7ef9923da33dee9121d930c06e01579a83fbe9" Nov 03 15:47:27.940541 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:27.940505 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj"] Nov 03 15:47:27.996607 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:27.996515 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-68kqj"] Nov 03 15:47:28.017898 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:28.016269 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct"] Nov 03 15:47:28.017898 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:28.016602 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" podUID="c5a211f3-7585-4727-bb04-b468c5c6e89f" containerName="istio-proxy" containerID="cri-o://3cf928d71649044f9cb06c332e9262390bed0094bfb9934a9a78624d88f36fb3" gracePeriod=30 Nov 03 15:47:28.677577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:28.677541 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" path="/var/lib/kubelet/pods/ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8/volumes" Nov 03 15:47:33.326793 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.326764 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:47:33.429461 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.429398 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a211f3-7585-4727-bb04-b468c5c6e89f-kube-api-access-pz7s6\" (UniqueName: \"kubernetes.io/projected/c5a211f3-7585-4727-bb04-b468c5c6e89f-kube-api-access-pz7s6\") pod \"c5a211f3-7585-4727-bb04-b468c5c6e89f\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " Nov 03 15:47:33.429654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.429486 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-envoy\") pod \"c5a211f3-7585-4727-bb04-b468c5c6e89f\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " Nov 03 15:47:33.429654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.429566 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-podinfo\") pod \"c5a211f3-7585-4727-bb04-b468c5c6e89f\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " Nov 03 15:47:33.429654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.429603 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a211f3-7585-4727-bb04-b468c5c6e89f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a211f3-7585-4727-bb04-b468c5c6e89f-istiod-ca-cert\") pod \"c5a211f3-7585-4727-bb04-b468c5c6e89f\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " Nov 03 15:47:33.429654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.429642 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-workload-certs\") pod \"c5a211f3-7585-4727-bb04-b468c5c6e89f\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " Nov 03 15:47:33.429948 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.429689 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-token\" (UniqueName: \"kubernetes.io/projected/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-token\") pod \"c5a211f3-7585-4727-bb04-b468c5c6e89f\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " Nov 03 15:47:33.429948 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.429761 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-workload-socket\") pod \"c5a211f3-7585-4727-bb04-b468c5c6e89f\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " Nov 03 15:47:33.429948 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.429862 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-data\") pod \"c5a211f3-7585-4727-bb04-b468c5c6e89f\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " Nov 03 15:47:33.432156 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.430800 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-credential-socket" pod "c5a211f3-7585-4727-bb04-b468c5c6e89f" (UID: "c5a211f3-7585-4727-bb04-b468c5c6e89f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:33.432156 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.431190 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-data" pod "c5a211f3-7585-4727-bb04-b468c5c6e89f" (UID: "c5a211f3-7585-4727-bb04-b468c5c6e89f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:33.432156 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.431767 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a211f3-7585-4727-bb04-b468c5c6e89f-istiod-ca-cert" pod "c5a211f3-7585-4727-bb04-b468c5c6e89f" (UID: "c5a211f3-7585-4727-bb04-b468c5c6e89f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:33.432156 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.432095 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-workload-certs" pod "c5a211f3-7585-4727-bb04-b468c5c6e89f" (UID: "c5a211f3-7585-4727-bb04-b468c5c6e89f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:33.445795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.429987 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-credential-socket\") pod \"c5a211f3-7585-4727-bb04-b468c5c6e89f\" (UID: \"c5a211f3-7585-4727-bb04-b468c5c6e89f\") " Nov 03 15:47:33.445795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.433357 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:33.445795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.433384 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:33.445795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.433409 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a211f3-7585-4727-bb04-b468c5c6e89f-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:33.445795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.433425 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:33.445795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.434781 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-workload-socket" pod "c5a211f3-7585-4727-bb04-b468c5c6e89f" (UID: "c5a211f3-7585-4727-bb04-b468c5c6e89f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:33.445795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.438139 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a211f3-7585-4727-bb04-b468c5c6e89f-kube-api-access-pz7s6" pod "c5a211f3-7585-4727-bb04-b468c5c6e89f" (UID: "c5a211f3-7585-4727-bb04-b468c5c6e89f"). InnerVolumeSpecName "kube-api-access-pz7s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:33.445795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.438219 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-podinfo" pod "c5a211f3-7585-4727-bb04-b468c5c6e89f" (UID: "c5a211f3-7585-4727-bb04-b468c5c6e89f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:47:33.445795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.444895 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-envoy" pod "c5a211f3-7585-4727-bb04-b468c5c6e89f" (UID: "c5a211f3-7585-4727-bb04-b468c5c6e89f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:33.464412 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.464375 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-token" pod "c5a211f3-7585-4727-bb04-b468c5c6e89f" (UID: "c5a211f3-7585-4727-bb04-b468c5c6e89f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:33.534485 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.534389 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:33.534485 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.534425 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:33.534485 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.534437 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:33.534485 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.534446 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz7s6\" (UniqueName: \"kubernetes.io/projected/c5a211f3-7585-4727-bb04-b468c5c6e89f-kube-api-access-pz7s6\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:33.534485 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.534460 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a211f3-7585-4727-bb04-b468c5c6e89f-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:33.776728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.774833 4290 generic.go:356] "Generic (PLEG): container finished" podID="c5a211f3-7585-4727-bb04-b468c5c6e89f" containerID="3cf928d71649044f9cb06c332e9262390bed0094bfb9934a9a78624d88f36fb3" exitCode=0 Nov 03 15:47:33.776728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.774938 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" event={"ID":"c5a211f3-7585-4727-bb04-b468c5c6e89f","Type":"ContainerDied","Data":"3cf928d71649044f9cb06c332e9262390bed0094bfb9934a9a78624d88f36fb3"} Nov 03 15:47:33.776728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.774987 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" event={"ID":"c5a211f3-7585-4727-bb04-b468c5c6e89f","Type":"ContainerDied","Data":"610afe3f82918349d4d76b4e81d50c2b357cae4dce682a2020edd3510f821854"} Nov 03 15:47:33.776728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.775011 4290 scope.go:122] "RemoveContainer" containerID="3cf928d71649044f9cb06c332e9262390bed0094bfb9934a9a78624d88f36fb3" Nov 03 15:47:33.776728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.775169 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct" Nov 03 15:47:33.822905 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.822878 4290 scope.go:122] "RemoveContainer" containerID="3cf928d71649044f9cb06c332e9262390bed0094bfb9934a9a78624d88f36fb3" Nov 03 15:47:33.823539 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:47:33.823512 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf928d71649044f9cb06c332e9262390bed0094bfb9934a9a78624d88f36fb3\": container with ID starting with 3cf928d71649044f9cb06c332e9262390bed0094bfb9934a9a78624d88f36fb3 not found: ID does not exist" containerID="3cf928d71649044f9cb06c332e9262390bed0094bfb9934a9a78624d88f36fb3" Nov 03 15:47:33.823729 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.823680 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf928d71649044f9cb06c332e9262390bed0094bfb9934a9a78624d88f36fb3"} err="failed to get container status \"3cf928d71649044f9cb06c332e9262390bed0094bfb9934a9a78624d88f36fb3\": rpc error: code = NotFound desc = could not find container \"3cf928d71649044f9cb06c332e9262390bed0094bfb9934a9a78624d88f36fb3\": container with ID starting with 3cf928d71649044f9cb06c332e9262390bed0094bfb9934a9a78624d88f36fb3 not found: ID does not exist" Nov 03 15:47:33.965576 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.965543 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct"] Nov 03 15:47:33.968934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:33.968891 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-fgmct"] Nov 03 15:47:34.713461 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:34.713418 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a211f3-7585-4727-bb04-b468c5c6e89f" path="/var/lib/kubelet/pods/c5a211f3-7585-4727-bb04-b468c5c6e89f/volumes" Nov 03 15:47:38.703639 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:47:38.703597 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:47:48.050944 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:48.050909 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd"] Nov 03 15:47:48.100511 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:48.100477 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b08a2daf-gwvfd"] Nov 03 15:47:48.685527 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:48.685486 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be2add4-64cd-435f-8cc2-69eca189384f" path="/var/lib/kubelet/pods/8be2add4-64cd-435f-8cc2-69eca189384f/volumes" Nov 03 15:47:57.050138 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:57.050055 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph"] Nov 03 15:47:57.099979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:57.099945 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddeebef3-7zlph"] Nov 03 15:47:58.053669 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:58.053635 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k"] Nov 03 15:47:58.100638 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:58.100606 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33e18c8-ntc4k"] Nov 03 15:47:58.677959 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:58.677929 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71993bd0-5109-43ca-a5d9-ab525546ced5" path="/var/lib/kubelet/pods/71993bd0-5109-43ca-a5d9-ab525546ced5/volumes" Nov 03 15:47:58.678362 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:47:58.678349 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d9bd51-c718-47a2-8aac-1f32315e1592" path="/var/lib/kubelet/pods/e0d9bd51-c718-47a2-8aac-1f32315e1592/volumes" Nov 03 15:48:08.704947 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:48:08.704860 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:48:09.482299 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482267 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8"] Nov 03 15:48:09.482745 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482731 4290 cpu_manager.go:414] "containerMap: removing container" podUID="5ce34d51-acb2-484d-acee-275b15ec6532" containerName="payload-pre-processing" Nov 03 15:48:09.482801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482747 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce34d51-acb2-484d-acee-275b15ec6532" containerName="payload-pre-processing" Nov 03 15:48:09.482801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482767 4290 cpu_manager.go:414] "containerMap: removing container" podUID="3f80f0c3-6c46-4763-bd3c-0b4737b40044" containerName="istio-proxy" Nov 03 15:48:09.482801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482773 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f80f0c3-6c46-4763-bd3c-0b4737b40044" containerName="istio-proxy" Nov 03 15:48:09.482801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482785 4290 cpu_manager.go:414] "containerMap: removing container" podUID="d04ac745-e80e-433c-a131-d61612494260" containerName="istio-proxy" Nov 03 15:48:09.482801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482790 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d04ac745-e80e-433c-a131-d61612494260" containerName="istio-proxy" Nov 03 15:48:09.482801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482798 4290 cpu_manager.go:414] "containerMap: removing container" podUID="ad3f085b-e44a-4f7a-a92e-065c19adf7d8" containerName="payload-processing" Nov 03 15:48:09.482801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482803 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3f085b-e44a-4f7a-a92e-065c19adf7d8" containerName="payload-processing" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482813 4290 cpu_manager.go:414] "containerMap: removing container" podUID="fbb5aae0-bbca-4f1f-83c3-875d0ffd9001" containerName="maas-api" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482817 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb5aae0-bbca-4f1f-83c3-875d0ffd9001" containerName="maas-api" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482826 4290 cpu_manager.go:414] "containerMap: removing container" podUID="c5a211f3-7585-4727-bb04-b468c5c6e89f" containerName="istio-proxy" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482830 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a211f3-7585-4727-bb04-b468c5c6e89f" containerName="istio-proxy" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482838 4290 cpu_manager.go:414] "containerMap: removing container" podUID="fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" containerName="istio-proxy" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482843 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" containerName="istio-proxy" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482860 4290 cpu_manager.go:414] "containerMap: removing container" podUID="14cc323b-4437-4c76-adf8-dc2a7dbd4da2" containerName="revoke-keys" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482864 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cc323b-4437-4c76-adf8-dc2a7dbd4da2" containerName="revoke-keys" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482872 4290 cpu_manager.go:414] "containerMap: removing container" podUID="76e830bb-3e77-42b6-9d03-d41e24955b37" containerName="payload-pre-processing" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482876 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e830bb-3e77-42b6-9d03-d41e24955b37" containerName="payload-pre-processing" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482882 4290 cpu_manager.go:414] "containerMap: removing container" podUID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" containerName="istio-proxy" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482886 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" containerName="istio-proxy" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482892 4290 cpu_manager.go:414] "containerMap: removing container" podUID="afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" containerName="istio-proxy" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482896 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" containerName="istio-proxy" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482906 4290 cpu_manager.go:414] "containerMap: removing container" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482911 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482976 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cc323b-4437-4c76-adf8-dc2a7dbd4da2" containerName="revoke-keys" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482986 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e830bb-3e77-42b6-9d03-d41e24955b37" containerName="payload-pre-processing" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482991 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e58dd3-dccf-45da-93fd-e4bbabf337db" containerName="istio-proxy" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.482996 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="afad8a84-b2c2-4ab6-8e5c-9d6f67779ec8" containerName="istio-proxy" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.483003 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee56237d-9d9c-4b6b-b3cb-4c37ad46f5e8" containerName="istio-proxy" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.483009 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce34d51-acb2-484d-acee-275b15ec6532" containerName="payload-pre-processing" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.483018 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f80f0c3-6c46-4763-bd3c-0b4737b40044" containerName="istio-proxy" Nov 03 15:48:09.483017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.483023 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="d04ac745-e80e-433c-a131-d61612494260" containerName="istio-proxy" Nov 03 15:48:09.483724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.483029 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3f085b-e44a-4f7a-a92e-065c19adf7d8" containerName="payload-processing" Nov 03 15:48:09.483724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.483034 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb5aae0-bbca-4f1f-83c3-875d0ffd9001" containerName="maas-api" Nov 03 15:48:09.483724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.483039 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a211f3-7585-4727-bb04-b468c5c6e89f" containerName="istio-proxy" Nov 03 15:48:09.483724 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.483045 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb1b2ea-09d4-4ce3-ab97-5d1c48c4c27c" containerName="istio-proxy" Nov 03 15:48:09.487400 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.487382 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" Nov 03 15:48:09.544828 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.544801 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8"] Nov 03 15:48:09.571098 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.571066 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e27d10df-774c-4fc9-a239-cf371d39c84d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32cf43e3-wwcx8\" (UID: \"e27d10df-774c-4fc9-a239-cf371d39c84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" Nov 03 15:48:09.571238 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.571134 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e27d10df-774c-4fc9-a239-cf371d39c84d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32cf43e3-wwcx8\" (UID: \"e27d10df-774c-4fc9-a239-cf371d39c84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" Nov 03 15:48:09.571238 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.571223 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kczz\" (UniqueName: \"kubernetes.io/projected/e27d10df-774c-4fc9-a239-cf371d39c84d-kube-api-access-8kczz\") pod \"maas-api-revoke-sub-32cf43e3-wwcx8\" (UID: \"e27d10df-774c-4fc9-a239-cf371d39c84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" Nov 03 15:48:09.590388 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.590363 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:48:09.590388 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.590379 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:48:09.590544 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.590389 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:48:09.590544 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.590373 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:48:09.672269 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.672236 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kczz\" (UniqueName: \"kubernetes.io/projected/e27d10df-774c-4fc9-a239-cf371d39c84d-kube-api-access-8kczz\") pod \"maas-api-revoke-sub-32cf43e3-wwcx8\" (UID: \"e27d10df-774c-4fc9-a239-cf371d39c84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" Nov 03 15:48:09.672446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.672315 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e27d10df-774c-4fc9-a239-cf371d39c84d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32cf43e3-wwcx8\" (UID: \"e27d10df-774c-4fc9-a239-cf371d39c84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" Nov 03 15:48:09.672446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.672358 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e27d10df-774c-4fc9-a239-cf371d39c84d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32cf43e3-wwcx8\" (UID: \"e27d10df-774c-4fc9-a239-cf371d39c84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" Nov 03 15:48:09.672919 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.672899 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e27d10df-774c-4fc9-a239-cf371d39c84d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32cf43e3-wwcx8\" (UID: \"e27d10df-774c-4fc9-a239-cf371d39c84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" Nov 03 15:48:09.674740 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.674716 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e27d10df-774c-4fc9-a239-cf371d39c84d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32cf43e3-wwcx8\" (UID: \"e27d10df-774c-4fc9-a239-cf371d39c84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" Nov 03 15:48:09.734591 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.734530 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kczz\" (UniqueName: \"kubernetes.io/projected/e27d10df-774c-4fc9-a239-cf371d39c84d-kube-api-access-8kczz\") pod \"maas-api-revoke-sub-32cf43e3-wwcx8\" (UID: \"e27d10df-774c-4fc9-a239-cf371d39c84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" Nov 03 15:48:09.800921 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:09.800892 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" Nov 03 15:48:10.180273 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:10.180246 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8"] Nov 03 15:48:10.181509 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:48:10.181480 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27d10df_774c_4fc9_a239_cf371d39c84d.slice/crio-6b09b30e04e45da8c1bfa7016e4d0f6e093298198a9bc6741e15d9c4c56929e1 WatchSource:0}: Error finding container 6b09b30e04e45da8c1bfa7016e4d0f6e093298198a9bc6741e15d9c4c56929e1: Status 404 returned error can't find the container with id 6b09b30e04e45da8c1bfa7016e4d0f6e093298198a9bc6741e15d9c4c56929e1 Nov 03 15:48:11.091010 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:11.090979 4290 generic.go:356] "Generic (PLEG): container finished" podID="e27d10df-774c-4fc9-a239-cf371d39c84d" containerID="a24b974848c7cd62706b0dd0a193abc79aeef2ad5628152d264271ea0c395ca4" exitCode=0 Nov 03 15:48:11.091378 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:11.091069 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" event={"ID":"e27d10df-774c-4fc9-a239-cf371d39c84d","Type":"ContainerDied","Data":"a24b974848c7cd62706b0dd0a193abc79aeef2ad5628152d264271ea0c395ca4"} Nov 03 15:48:11.091378 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:11.091125 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" event={"ID":"e27d10df-774c-4fc9-a239-cf371d39c84d","Type":"ContainerStarted","Data":"6b09b30e04e45da8c1bfa7016e4d0f6e093298198a9bc6741e15d9c4c56929e1"} Nov 03 15:48:12.232654 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:12.232630 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" Nov 03 15:48:12.301690 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:12.301660 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e27d10df-774c-4fc9-a239-cf371d39c84d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e27d10df-774c-4fc9-a239-cf371d39c84d-maas-api-xks-ca\") pod \"e27d10df-774c-4fc9-a239-cf371d39c84d\" (UID: \"e27d10df-774c-4fc9-a239-cf371d39c84d\") " Nov 03 15:48:12.301848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:12.301815 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e27d10df-774c-4fc9-a239-cf371d39c84d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e27d10df-774c-4fc9-a239-cf371d39c84d-maas-api-openshift-ca\") pod \"e27d10df-774c-4fc9-a239-cf371d39c84d\" (UID: \"e27d10df-774c-4fc9-a239-cf371d39c84d\") " Nov 03 15:48:12.301907 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:12.301881 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27d10df-774c-4fc9-a239-cf371d39c84d-kube-api-access-8kczz\" (UniqueName: \"kubernetes.io/projected/e27d10df-774c-4fc9-a239-cf371d39c84d-kube-api-access-8kczz\") pod \"e27d10df-774c-4fc9-a239-cf371d39c84d\" (UID: \"e27d10df-774c-4fc9-a239-cf371d39c84d\") " Nov 03 15:48:12.302176 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:12.302152 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27d10df-774c-4fc9-a239-cf371d39c84d-maas-api-openshift-ca" pod "e27d10df-774c-4fc9-a239-cf371d39c84d" (UID: "e27d10df-774c-4fc9-a239-cf371d39c84d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:48:12.303858 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:12.303835 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e27d10df-774c-4fc9-a239-cf371d39c84d-maas-api-xks-ca" pod "e27d10df-774c-4fc9-a239-cf371d39c84d" (UID: "e27d10df-774c-4fc9-a239-cf371d39c84d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:48:12.304058 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:12.304035 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27d10df-774c-4fc9-a239-cf371d39c84d-kube-api-access-8kczz" pod "e27d10df-774c-4fc9-a239-cf371d39c84d" (UID: "e27d10df-774c-4fc9-a239-cf371d39c84d"). InnerVolumeSpecName "kube-api-access-8kczz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:48:12.403040 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:12.403013 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e27d10df-774c-4fc9-a239-cf371d39c84d-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:12.403040 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:12.403039 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e27d10df-774c-4fc9-a239-cf371d39c84d-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:12.403233 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:12.403049 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kczz\" (UniqueName: \"kubernetes.io/projected/e27d10df-774c-4fc9-a239-cf371d39c84d-kube-api-access-8kczz\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:13.107610 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:13.107527 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" event={"ID":"e27d10df-774c-4fc9-a239-cf371d39c84d","Type":"ContainerDied","Data":"6b09b30e04e45da8c1bfa7016e4d0f6e093298198a9bc6741e15d9c4c56929e1"} Nov 03 15:48:13.107610 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:13.107550 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8" Nov 03 15:48:13.107610 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:13.107605 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b09b30e04e45da8c1bfa7016e4d0f6e093298198a9bc6741e15d9c4c56929e1" Nov 03 15:48:13.234338 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:13.234308 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:48:13.236961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:13.236940 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:48:13.245618 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:13.245599 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:48:13.248475 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:13.248457 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:48:17.684242 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:17.684209 4290 scope.go:122] "RemoveContainer" containerID="4ea9f0a5b010122891bfdbf8144cc663018c5f4a72a86d0e9a6b8afaf6816f62" Nov 03 15:48:17.694924 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:17.694900 4290 scope.go:122] "RemoveContainer" containerID="2030d386f300888cbece7fed695c7da98d17add9bb2f8cefa30241a74eaff9b2" Nov 03 15:48:20.793966 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.793928 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z"] Nov 03 15:48:20.794630 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.794612 4290 cpu_manager.go:414] "containerMap: removing container" podUID="e27d10df-774c-4fc9-a239-cf371d39c84d" containerName="revoke-keys" Nov 03 15:48:20.794682 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.794633 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27d10df-774c-4fc9-a239-cf371d39c84d" containerName="revoke-keys" Nov 03 15:48:20.794752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.794746 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27d10df-774c-4fc9-a239-cf371d39c84d" containerName="revoke-keys" Nov 03 15:48:20.799677 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.799658 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" Nov 03 15:48:20.894975 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.894946 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z\" (UID: \"cad6547e-79fd-4d3a-a97f-9e17bf8614a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" Nov 03 15:48:20.895151 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.895095 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2zjg\" (UniqueName: \"kubernetes.io/projected/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-kube-api-access-w2zjg\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z\" (UID: \"cad6547e-79fd-4d3a-a97f-9e17bf8614a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" Nov 03 15:48:20.895218 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.895194 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z\" (UID: \"cad6547e-79fd-4d3a-a97f-9e17bf8614a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" Nov 03 15:48:20.905024 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.905000 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:48:20.905143 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.905048 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:48:20.905439 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.905425 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:48:20.905474 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.905426 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:48:20.908443 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.908425 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z"] Nov 03 15:48:20.996441 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.996408 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z\" (UID: \"cad6547e-79fd-4d3a-a97f-9e17bf8614a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" Nov 03 15:48:20.996628 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.996555 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2zjg\" (UniqueName: \"kubernetes.io/projected/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-kube-api-access-w2zjg\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z\" (UID: \"cad6547e-79fd-4d3a-a97f-9e17bf8614a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" Nov 03 15:48:20.996628 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.996617 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z\" (UID: \"cad6547e-79fd-4d3a-a97f-9e17bf8614a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" Nov 03 15:48:20.997364 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.997344 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z\" (UID: \"cad6547e-79fd-4d3a-a97f-9e17bf8614a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" Nov 03 15:48:20.999021 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:20.999003 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z\" (UID: \"cad6547e-79fd-4d3a-a97f-9e17bf8614a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" Nov 03 15:48:21.061137 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:21.061062 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2zjg\" (UniqueName: \"kubernetes.io/projected/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-kube-api-access-w2zjg\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z\" (UID: \"cad6547e-79fd-4d3a-a97f-9e17bf8614a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" Nov 03 15:48:21.114115 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:21.114085 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" Nov 03 15:48:21.522510 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:21.522479 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z"] Nov 03 15:48:21.523436 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:48:21.523398 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad6547e_79fd_4d3a_a97f_9e17bf8614a5.slice/crio-cdf44959639c88ee2528c837b41e29c62b1ad73214159c9224e71a5ae9704fdd WatchSource:0}: Error finding container cdf44959639c88ee2528c837b41e29c62b1ad73214159c9224e71a5ae9704fdd: Status 404 returned error can't find the container with id cdf44959639c88ee2528c837b41e29c62b1ad73214159c9224e71a5ae9704fdd Nov 03 15:48:22.179383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:22.179344 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" event={"ID":"cad6547e-79fd-4d3a-a97f-9e17bf8614a5","Type":"ContainerStarted","Data":"b6ce3f82f3eb29dd8b3c027522dca97c2a3aaae2ec2d66fb242005e2938bb139"} Nov 03 15:48:22.179935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:22.179407 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" event={"ID":"cad6547e-79fd-4d3a-a97f-9e17bf8614a5","Type":"ContainerStarted","Data":"cdf44959639c88ee2528c837b41e29c62b1ad73214159c9224e71a5ae9704fdd"} Nov 03 15:48:22.348057 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:22.348002 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" podStartSLOduration=2.347983944 podStartE2EDuration="2.347983944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:48:22.299380877 +0000 UTC m=+2409.730338683" watchObservedRunningTime="2026-11-03 15:48:22.347983944 +0000 UTC m=+2409.778941758" Nov 03 15:48:23.187759 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:23.187722 4290 generic.go:356] "Generic (PLEG): container finished" podID="cad6547e-79fd-4d3a-a97f-9e17bf8614a5" containerID="b6ce3f82f3eb29dd8b3c027522dca97c2a3aaae2ec2d66fb242005e2938bb139" exitCode=0 Nov 03 15:48:23.188226 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:23.187795 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" event={"ID":"cad6547e-79fd-4d3a-a97f-9e17bf8614a5","Type":"ContainerDied","Data":"b6ce3f82f3eb29dd8b3c027522dca97c2a3aaae2ec2d66fb242005e2938bb139"} Nov 03 15:48:24.719414 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:24.719118 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" Nov 03 15:48:24.777127 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:24.776079 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-maas-api-openshift-ca\") pod \"cad6547e-79fd-4d3a-a97f-9e17bf8614a5\" (UID: \"cad6547e-79fd-4d3a-a97f-9e17bf8614a5\") " Nov 03 15:48:24.777127 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:24.776213 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-maas-api-xks-ca\") pod \"cad6547e-79fd-4d3a-a97f-9e17bf8614a5\" (UID: \"cad6547e-79fd-4d3a-a97f-9e17bf8614a5\") " Nov 03 15:48:24.777127 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:24.776343 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-kube-api-access-w2zjg\" (UniqueName: \"kubernetes.io/projected/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-kube-api-access-w2zjg\") pod \"cad6547e-79fd-4d3a-a97f-9e17bf8614a5\" (UID: \"cad6547e-79fd-4d3a-a97f-9e17bf8614a5\") " Nov 03 15:48:24.777974 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:24.777947 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-maas-api-openshift-ca" pod "cad6547e-79fd-4d3a-a97f-9e17bf8614a5" (UID: "cad6547e-79fd-4d3a-a97f-9e17bf8614a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:48:24.823031 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:24.822732 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-maas-api-xks-ca" pod "cad6547e-79fd-4d3a-a97f-9e17bf8614a5" (UID: "cad6547e-79fd-4d3a-a97f-9e17bf8614a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:48:24.823290 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:24.823244 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-kube-api-access-w2zjg" pod "cad6547e-79fd-4d3a-a97f-9e17bf8614a5" (UID: "cad6547e-79fd-4d3a-a97f-9e17bf8614a5"). InnerVolumeSpecName "kube-api-access-w2zjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:48:24.923788 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:24.921998 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:24.923788 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:24.922038 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2zjg\" (UniqueName: \"kubernetes.io/projected/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-kube-api-access-w2zjg\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:24.923788 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:24.922054 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cad6547e-79fd-4d3a-a97f-9e17bf8614a5-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:25.206068 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:25.205968 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f"] Nov 03 15:48:25.226505 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:25.226463 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" event={"ID":"cad6547e-79fd-4d3a-a97f-9e17bf8614a5","Type":"ContainerDied","Data":"cdf44959639c88ee2528c837b41e29c62b1ad73214159c9224e71a5ae9704fdd"} Nov 03 15:48:25.226670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:25.226523 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdf44959639c88ee2528c837b41e29c62b1ad73214159c9224e71a5ae9704fdd" Nov 03 15:48:25.226670 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:25.226592 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z" Nov 03 15:48:25.560074 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:25.559979 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pvhdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:48:25.560363 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:25.560330 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" podUID="c56971b8-db79-464a-bdaa-1a1e950f7c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:48:32.293390 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:32.293352 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d"] Nov 03 15:48:32.403130 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:32.403088 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8"] Nov 03 15:48:37.294386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:37.294329 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" podUID="d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36" containerName="payload-processing" containerID="cri-o://0dbdb933f7faec5c73a8f35b077def64251dcc0b7bcfc7002b0cb136eec89ff0" gracePeriod=25 Nov 03 15:48:37.403976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:37.403915 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" podUID="0866b376-9269-41cf-aa1a-d29d433718e7" containerName="payload-pre-processing" containerID="cri-o://f2bfaff1cff0fb968300f103fa969cf5bf04be032eab2331a9f2404e2ff05310" gracePeriod=25 Nov 03 15:48:38.703222 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:48:38.703188 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:48:38.789824 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.789782 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn"] Nov 03 15:48:38.790549 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.790533 4290 cpu_manager.go:414] "containerMap: removing container" podUID="cad6547e-79fd-4d3a-a97f-9e17bf8614a5" containerName="revoke-keys" Nov 03 15:48:38.790598 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.790552 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad6547e-79fd-4d3a-a97f-9e17bf8614a5" containerName="revoke-keys" Nov 03 15:48:38.790637 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.790632 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad6547e-79fd-4d3a-a97f-9e17bf8614a5" containerName="revoke-keys" Nov 03 15:48:38.794177 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.794160 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" Nov 03 15:48:38.891451 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.891406 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f7v4\" (UniqueName: \"kubernetes.io/projected/5e47873e-a311-4b93-ac35-a1b02e509efd-kube-api-access-9f7v4\") pod \"maas-api-revoke-sub-8a257d6e-pvvdn\" (UID: \"5e47873e-a311-4b93-ac35-a1b02e509efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" Nov 03 15:48:38.891613 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.891519 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e47873e-a311-4b93-ac35-a1b02e509efd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a257d6e-pvvdn\" (UID: \"5e47873e-a311-4b93-ac35-a1b02e509efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" Nov 03 15:48:38.891694 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.891675 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e47873e-a311-4b93-ac35-a1b02e509efd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a257d6e-pvvdn\" (UID: \"5e47873e-a311-4b93-ac35-a1b02e509efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" Nov 03 15:48:38.896975 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.896945 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:48:38.900591 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.900561 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn"] Nov 03 15:48:38.935762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.935683 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:48:38.935762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.935686 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:48:38.935993 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.935914 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:48:38.993064 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.992976 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e47873e-a311-4b93-ac35-a1b02e509efd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a257d6e-pvvdn\" (UID: \"5e47873e-a311-4b93-ac35-a1b02e509efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" Nov 03 15:48:38.993064 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.993023 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f7v4\" (UniqueName: \"kubernetes.io/projected/5e47873e-a311-4b93-ac35-a1b02e509efd-kube-api-access-9f7v4\") pod \"maas-api-revoke-sub-8a257d6e-pvvdn\" (UID: \"5e47873e-a311-4b93-ac35-a1b02e509efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" Nov 03 15:48:38.993284 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.993077 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e47873e-a311-4b93-ac35-a1b02e509efd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a257d6e-pvvdn\" (UID: \"5e47873e-a311-4b93-ac35-a1b02e509efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" Nov 03 15:48:38.993777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.993746 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e47873e-a311-4b93-ac35-a1b02e509efd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a257d6e-pvvdn\" (UID: \"5e47873e-a311-4b93-ac35-a1b02e509efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" Nov 03 15:48:38.995578 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:38.995555 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e47873e-a311-4b93-ac35-a1b02e509efd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a257d6e-pvvdn\" (UID: \"5e47873e-a311-4b93-ac35-a1b02e509efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" Nov 03 15:48:39.055192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:39.055156 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f7v4\" (UniqueName: \"kubernetes.io/projected/5e47873e-a311-4b93-ac35-a1b02e509efd-kube-api-access-9f7v4\") pod \"maas-api-revoke-sub-8a257d6e-pvvdn\" (UID: \"5e47873e-a311-4b93-ac35-a1b02e509efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" Nov 03 15:48:39.108569 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:39.108535 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" Nov 03 15:48:39.510227 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:39.510198 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn"] Nov 03 15:48:39.512201 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:48:39.512174 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e47873e_a311_4b93_ac35_a1b02e509efd.slice/crio-84f592ce80932f81309f3c439bdaac8bba8137e3798876f2c2b1edeaf238ad07 WatchSource:0}: Error finding container 84f592ce80932f81309f3c439bdaac8bba8137e3798876f2c2b1edeaf238ad07: Status 404 returned error can't find the container with id 84f592ce80932f81309f3c439bdaac8bba8137e3798876f2c2b1edeaf238ad07 Nov 03 15:48:40.235315 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.235274 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" containerID="cri-o://8705d5224f9a26868f51bd2dbabbf4cfb7e8fd8061e592dfb36338ae2345663e" gracePeriod=45 Nov 03 15:48:40.335253 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.335213 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" event={"ID":"5e47873e-a311-4b93-ac35-a1b02e509efd","Type":"ContainerStarted","Data":"dcec3bfe6f55469972c4a8bfef4f7ae07d8c20d35025c6c50a1a3d802b6b1fdb"} Nov 03 15:48:40.335432 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.335269 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" event={"ID":"5e47873e-a311-4b93-ac35-a1b02e509efd","Type":"ContainerStarted","Data":"84f592ce80932f81309f3c439bdaac8bba8137e3798876f2c2b1edeaf238ad07"} Nov 03 15:48:40.454098 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.454045 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" podStartSLOduration=2.454029439 podStartE2EDuration="2.454029439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:48:40.451989544 +0000 UTC m=+2427.882947350" watchObservedRunningTime="2026-11-03 15:48:40.454029439 +0000 UTC m=+2427.884987245" Nov 03 15:48:40.525196 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.525162 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:48:40.613676 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.613644 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed595ae7-8d05-42d9-84e8-4a370338dcb8-kube-api-access-sq8mf\" (UniqueName: \"kubernetes.io/projected/ed595ae7-8d05-42d9-84e8-4a370338dcb8-kube-api-access-sq8mf\") pod \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " Nov 03 15:48:40.613895 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.613743 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed595ae7-8d05-42d9-84e8-4a370338dcb8-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed595ae7-8d05-42d9-84e8-4a370338dcb8-tls-certs\") pod \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " Nov 03 15:48:40.613895 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.613783 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-home\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-home\") pod \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " Nov 03 15:48:40.613895 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.613825 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-tmp-dir\") pod \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " Nov 03 15:48:40.613895 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.613889 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-model-cache\") pod \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " Nov 03 15:48:40.614112 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.614025 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-dshm\") pod \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\" (UID: \"ed595ae7-8d05-42d9-84e8-4a370338dcb8\") " Nov 03 15:48:40.614163 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.614116 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-tmp-dir" pod "ed595ae7-8d05-42d9-84e8-4a370338dcb8" (UID: "ed595ae7-8d05-42d9-84e8-4a370338dcb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:48:40.614163 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.614101 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-home" pod "ed595ae7-8d05-42d9-84e8-4a370338dcb8" (UID: "ed595ae7-8d05-42d9-84e8-4a370338dcb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:48:40.614262 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.614152 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-model-cache" pod "ed595ae7-8d05-42d9-84e8-4a370338dcb8" (UID: "ed595ae7-8d05-42d9-84e8-4a370338dcb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:48:40.614594 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.614572 4290 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-home\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:40.614727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.614595 4290 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-tmp-dir\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:40.614727 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.614609 4290 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-model-cache\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:40.616179 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.616151 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed595ae7-8d05-42d9-84e8-4a370338dcb8-tls-certs" pod "ed595ae7-8d05-42d9-84e8-4a370338dcb8" (UID: "ed595ae7-8d05-42d9-84e8-4a370338dcb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:48:40.616267 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.616240 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-dshm" pod "ed595ae7-8d05-42d9-84e8-4a370338dcb8" (UID: "ed595ae7-8d05-42d9-84e8-4a370338dcb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:48:40.616345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.616268 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed595ae7-8d05-42d9-84e8-4a370338dcb8-kube-api-access-sq8mf" pod "ed595ae7-8d05-42d9-84e8-4a370338dcb8" (UID: "ed595ae7-8d05-42d9-84e8-4a370338dcb8"). InnerVolumeSpecName "kube-api-access-sq8mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:48:40.715515 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.715475 4290 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed595ae7-8d05-42d9-84e8-4a370338dcb8-dshm\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:40.715515 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.715507 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq8mf\" (UniqueName: \"kubernetes.io/projected/ed595ae7-8d05-42d9-84e8-4a370338dcb8-kube-api-access-sq8mf\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:40.715515 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:40.715518 4290 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed595ae7-8d05-42d9-84e8-4a370338dcb8-tls-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:41.343821 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:41.343788 4290 generic.go:356] "Generic (PLEG): container finished" podID="5e47873e-a311-4b93-ac35-a1b02e509efd" containerID="dcec3bfe6f55469972c4a8bfef4f7ae07d8c20d35025c6c50a1a3d802b6b1fdb" exitCode=0 Nov 03 15:48:41.344261 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:41.343881 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" event={"ID":"5e47873e-a311-4b93-ac35-a1b02e509efd","Type":"ContainerDied","Data":"dcec3bfe6f55469972c4a8bfef4f7ae07d8c20d35025c6c50a1a3d802b6b1fdb"} Nov 03 15:48:41.345381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:41.345355 4290 generic.go:356] "Generic (PLEG): container finished" podID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerID="8705d5224f9a26868f51bd2dbabbf4cfb7e8fd8061e592dfb36338ae2345663e" exitCode=0 Nov 03 15:48:41.345503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:41.345380 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" event={"ID":"ed595ae7-8d05-42d9-84e8-4a370338dcb8","Type":"ContainerDied","Data":"8705d5224f9a26868f51bd2dbabbf4cfb7e8fd8061e592dfb36338ae2345663e"} Nov 03 15:48:41.345503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:41.345421 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" event={"ID":"ed595ae7-8d05-42d9-84e8-4a370338dcb8","Type":"ContainerDied","Data":"96397a6887cfe10725770b930940877d7229b267f4ed0cf0499b3ed6d409d310"} Nov 03 15:48:41.345503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:41.345426 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f" Nov 03 15:48:41.345503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:41.345444 4290 scope.go:122] "RemoveContainer" containerID="8705d5224f9a26868f51bd2dbabbf4cfb7e8fd8061e592dfb36338ae2345663e" Nov 03 15:48:41.356618 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:41.356589 4290 scope.go:122] "RemoveContainer" containerID="8705d5224f9a26868f51bd2dbabbf4cfb7e8fd8061e592dfb36338ae2345663e" Nov 03 15:48:41.356977 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:48:41.356956 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8705d5224f9a26868f51bd2dbabbf4cfb7e8fd8061e592dfb36338ae2345663e\": container with ID starting with 8705d5224f9a26868f51bd2dbabbf4cfb7e8fd8061e592dfb36338ae2345663e not found: ID does not exist" containerID="8705d5224f9a26868f51bd2dbabbf4cfb7e8fd8061e592dfb36338ae2345663e" Nov 03 15:48:41.357055 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:41.356987 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8705d5224f9a26868f51bd2dbabbf4cfb7e8fd8061e592dfb36338ae2345663e"} err="failed to get container status \"8705d5224f9a26868f51bd2dbabbf4cfb7e8fd8061e592dfb36338ae2345663e\": rpc error: code = NotFound desc = could not find container \"8705d5224f9a26868f51bd2dbabbf4cfb7e8fd8061e592dfb36338ae2345663e\": container with ID starting with 8705d5224f9a26868f51bd2dbabbf4cfb7e8fd8061e592dfb36338ae2345663e not found: ID does not exist" Nov 03 15:48:41.574072 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:41.574033 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f"] Nov 03 15:48:41.625620 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:41.625582 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/distinct-w3-8b841a-kserve-7cddc66746-t6g4f"] Nov 03 15:48:42.520286 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:42.520259 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" Nov 03 15:48:42.636693 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:42.636638 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e47873e-a311-4b93-ac35-a1b02e509efd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e47873e-a311-4b93-ac35-a1b02e509efd-maas-api-openshift-ca\") pod \"5e47873e-a311-4b93-ac35-a1b02e509efd\" (UID: \"5e47873e-a311-4b93-ac35-a1b02e509efd\") " Nov 03 15:48:42.636946 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:42.636730 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e47873e-a311-4b93-ac35-a1b02e509efd-kube-api-access-9f7v4\" (UniqueName: \"kubernetes.io/projected/5e47873e-a311-4b93-ac35-a1b02e509efd-kube-api-access-9f7v4\") pod \"5e47873e-a311-4b93-ac35-a1b02e509efd\" (UID: \"5e47873e-a311-4b93-ac35-a1b02e509efd\") " Nov 03 15:48:42.636946 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:42.636764 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e47873e-a311-4b93-ac35-a1b02e509efd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e47873e-a311-4b93-ac35-a1b02e509efd-maas-api-xks-ca\") pod \"5e47873e-a311-4b93-ac35-a1b02e509efd\" (UID: \"5e47873e-a311-4b93-ac35-a1b02e509efd\") " Nov 03 15:48:42.637091 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:42.637071 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e47873e-a311-4b93-ac35-a1b02e509efd-maas-api-openshift-ca" pod "5e47873e-a311-4b93-ac35-a1b02e509efd" (UID: "5e47873e-a311-4b93-ac35-a1b02e509efd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:48:42.637367 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:42.637333 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e47873e-a311-4b93-ac35-a1b02e509efd-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:42.640951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:42.640909 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e47873e-a311-4b93-ac35-a1b02e509efd-maas-api-xks-ca" pod "5e47873e-a311-4b93-ac35-a1b02e509efd" (UID: "5e47873e-a311-4b93-ac35-a1b02e509efd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:48:42.641077 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:42.641013 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e47873e-a311-4b93-ac35-a1b02e509efd-kube-api-access-9f7v4" pod "5e47873e-a311-4b93-ac35-a1b02e509efd" (UID: "5e47873e-a311-4b93-ac35-a1b02e509efd"). InnerVolumeSpecName "kube-api-access-9f7v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:48:42.680031 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:42.679996 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" path="/var/lib/kubelet/pods/ed595ae7-8d05-42d9-84e8-4a370338dcb8/volumes" Nov 03 15:48:42.738971 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:42.738939 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f7v4\" (UniqueName: \"kubernetes.io/projected/5e47873e-a311-4b93-ac35-a1b02e509efd-kube-api-access-9f7v4\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:42.738971 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:42.738968 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e47873e-a311-4b93-ac35-a1b02e509efd-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:43.363418 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:43.363385 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" Nov 03 15:48:43.363621 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:43.363416 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn" event={"ID":"5e47873e-a311-4b93-ac35-a1b02e509efd","Type":"ContainerDied","Data":"84f592ce80932f81309f3c439bdaac8bba8137e3798876f2c2b1edeaf238ad07"} Nov 03 15:48:43.363621 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:43.363462 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84f592ce80932f81309f3c439bdaac8bba8137e3798876f2c2b1edeaf238ad07" Nov 03 15:48:46.324834 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:46.324789 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" podUID="d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:48:46.325332 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:46.324892 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" podUID="0866b376-9269-41cf-aa1a-d29d433718e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:48:56.325126 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:56.325079 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" podUID="0866b376-9269-41cf-aa1a-d29d433718e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:48:56.325126 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:56.325105 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" podUID="d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:48:56.482506 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:56.482475 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z"] Nov 03 15:48:56.492061 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:56.492027 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b841a-l6s2z"] Nov 03 15:48:56.678435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:56.678403 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad6547e-79fd-4d3a-a97f-9e17bf8614a5" path="/var/lib/kubelet/pods/cad6547e-79fd-4d3a-a97f-9e17bf8614a5/volumes" Nov 03 15:48:57.496388 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.496142 4290 generic.go:356] "Generic (PLEG): container finished" podID="d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36" containerID="0dbdb933f7faec5c73a8f35b077def64251dcc0b7bcfc7002b0cb136eec89ff0" exitCode=0 Nov 03 15:48:57.496388 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.496357 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" event={"ID":"d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36","Type":"ContainerDied","Data":"0dbdb933f7faec5c73a8f35b077def64251dcc0b7bcfc7002b0cb136eec89ff0"} Nov 03 15:48:57.558517 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.558446 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" Nov 03 15:48:57.595360 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.595328 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36-plugins-config-volume\") pod \"d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36\" (UID: \"d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36\") " Nov 03 15:48:57.595549 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.595428 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36-kube-api-access-lxxpt\" (UniqueName: \"kubernetes.io/projected/d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36-kube-api-access-lxxpt\") pod \"d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36\" (UID: \"d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36\") " Nov 03 15:48:57.595733 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.595684 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36-plugins-config-volume" pod "d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36" (UID: "d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:48:57.595913 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.595895 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:57.597806 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.597774 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36-kube-api-access-lxxpt" pod "d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36" (UID: "d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36"). InnerVolumeSpecName "kube-api-access-lxxpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:48:57.656805 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.656781 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" Nov 03 15:48:57.696339 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.696302 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0866b376-9269-41cf-aa1a-d29d433718e7-kube-api-access-4tzhh\" (UniqueName: \"kubernetes.io/projected/0866b376-9269-41cf-aa1a-d29d433718e7-kube-api-access-4tzhh\") pod \"0866b376-9269-41cf-aa1a-d29d433718e7\" (UID: \"0866b376-9269-41cf-aa1a-d29d433718e7\") " Nov 03 15:48:57.696510 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.696473 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0866b376-9269-41cf-aa1a-d29d433718e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0866b376-9269-41cf-aa1a-d29d433718e7-plugins-config-volume\") pod \"0866b376-9269-41cf-aa1a-d29d433718e7\" (UID: \"0866b376-9269-41cf-aa1a-d29d433718e7\") " Nov 03 15:48:57.696849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.696830 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxxpt\" (UniqueName: \"kubernetes.io/projected/d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36-kube-api-access-lxxpt\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:57.696849 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.696830 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0866b376-9269-41cf-aa1a-d29d433718e7-plugins-config-volume" pod "0866b376-9269-41cf-aa1a-d29d433718e7" (UID: "0866b376-9269-41cf-aa1a-d29d433718e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:48:57.698691 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.698664 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0866b376-9269-41cf-aa1a-d29d433718e7-kube-api-access-4tzhh" pod "0866b376-9269-41cf-aa1a-d29d433718e7" (UID: "0866b376-9269-41cf-aa1a-d29d433718e7"). InnerVolumeSpecName "kube-api-access-4tzhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:48:57.798158 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.798129 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0866b376-9269-41cf-aa1a-d29d433718e7-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:57.798158 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:57.798156 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tzhh\" (UniqueName: \"kubernetes.io/projected/0866b376-9269-41cf-aa1a-d29d433718e7-kube-api-access-4tzhh\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:58.504960 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:58.504931 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" Nov 03 15:48:58.504960 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:58.504953 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d" event={"ID":"d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36","Type":"ContainerDied","Data":"c1075121c351eb0f4bfb6ec043fd5e4e27f2c001ac35fef32549f71a9f832556"} Nov 03 15:48:58.505506 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:58.505030 4290 scope.go:122] "RemoveContainer" containerID="0dbdb933f7faec5c73a8f35b077def64251dcc0b7bcfc7002b0cb136eec89ff0" Nov 03 15:48:58.507531 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:58.507430 4290 generic.go:356] "Generic (PLEG): container finished" podID="0866b376-9269-41cf-aa1a-d29d433718e7" containerID="f2bfaff1cff0fb968300f103fa969cf5bf04be032eab2331a9f2404e2ff05310" exitCode=0 Nov 03 15:48:58.507850 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:58.507797 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" event={"ID":"0866b376-9269-41cf-aa1a-d29d433718e7","Type":"ContainerDied","Data":"f2bfaff1cff0fb968300f103fa969cf5bf04be032eab2331a9f2404e2ff05310"} Nov 03 15:48:58.508054 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:58.508023 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" Nov 03 15:48:58.508295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:58.508237 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8" event={"ID":"0866b376-9269-41cf-aa1a-d29d433718e7","Type":"ContainerDied","Data":"8061d19a3a259c32af693896005dd378139ad60b8f5f3f4280de68a5a0633159"} Nov 03 15:48:58.523778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:58.523749 4290 scope.go:122] "RemoveContainer" containerID="f2bfaff1cff0fb968300f103fa969cf5bf04be032eab2331a9f2404e2ff05310" Nov 03 15:48:58.553352 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:58.553325 4290 scope.go:122] "RemoveContainer" containerID="f2bfaff1cff0fb968300f103fa969cf5bf04be032eab2331a9f2404e2ff05310" Nov 03 15:48:58.553735 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:48:58.553711 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2bfaff1cff0fb968300f103fa969cf5bf04be032eab2331a9f2404e2ff05310\": container with ID starting with f2bfaff1cff0fb968300f103fa969cf5bf04be032eab2331a9f2404e2ff05310 not found: ID does not exist" containerID="f2bfaff1cff0fb968300f103fa969cf5bf04be032eab2331a9f2404e2ff05310" Nov 03 15:48:58.553822 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:58.553743 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2bfaff1cff0fb968300f103fa969cf5bf04be032eab2331a9f2404e2ff05310"} err="failed to get container status \"f2bfaff1cff0fb968300f103fa969cf5bf04be032eab2331a9f2404e2ff05310\": rpc error: code = NotFound desc = could not find container \"f2bfaff1cff0fb968300f103fa969cf5bf04be032eab2331a9f2404e2ff05310\": container with ID starting with f2bfaff1cff0fb968300f103fa969cf5bf04be032eab2331a9f2404e2ff05310 not found: ID does not exist" Nov 03 15:48:58.677141 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:58.677108 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8"] Nov 03 15:48:58.721975 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:58.721938 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8b841a-99d59fcf5-5mpp8"] Nov 03 15:48:58.831299 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:58.831264 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d"] Nov 03 15:48:58.845473 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:48:58.845441 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8b841a-7d89976cfc-sz58d"] Nov 03 15:49:00.678012 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:00.677978 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0866b376-9269-41cf-aa1a-d29d433718e7" path="/var/lib/kubelet/pods/0866b376-9269-41cf-aa1a-d29d433718e7/volumes" Nov 03 15:49:00.678411 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:00.678366 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36" path="/var/lib/kubelet/pods/d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36/volumes" Nov 03 15:49:05.978640 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:05.978599 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl"] Nov 03 15:49:05.979265 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:05.978901 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" containerID="cri-o://946748a29710d727cba118f7cda64bdc989bde662f7da4e72167a1b11fe0000a" gracePeriod=30 Nov 03 15:49:08.703083 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:49:08.702969 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:49:11.242423 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.242398 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:49:11.353099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.353001 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f213cf23-67be-4699-828e-23a0a966c12a-istio-token\" (UniqueName: \"kubernetes.io/projected/f213cf23-67be-4699-828e-23a0a966c12a-istio-token\") pod \"f213cf23-67be-4699-828e-23a0a966c12a\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " Nov 03 15:49:11.353099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.353042 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-istio-data\") pod \"f213cf23-67be-4699-828e-23a0a966c12a\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " Nov 03 15:49:11.353099 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.353074 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-credential-socket\") pod \"f213cf23-67be-4699-828e-23a0a966c12a\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " Nov 03 15:49:11.353390 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.353159 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-workload-certs\") pod \"f213cf23-67be-4699-828e-23a0a966c12a\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " Nov 03 15:49:11.353390 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.353222 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f213cf23-67be-4699-828e-23a0a966c12a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f213cf23-67be-4699-828e-23a0a966c12a-istiod-ca-cert\") pod \"f213cf23-67be-4699-828e-23a0a966c12a\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " Nov 03 15:49:11.353390 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.353308 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f213cf23-67be-4699-828e-23a0a966c12a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f213cf23-67be-4699-828e-23a0a966c12a-istio-podinfo\") pod \"f213cf23-67be-4699-828e-23a0a966c12a\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " Nov 03 15:49:11.353390 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.353339 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-istio-envoy\") pod \"f213cf23-67be-4699-828e-23a0a966c12a\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " Nov 03 15:49:11.353647 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.353391 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-credential-socket" pod "f213cf23-67be-4699-828e-23a0a966c12a" (UID: "f213cf23-67be-4699-828e-23a0a966c12a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:49:11.353647 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.353421 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-istio-data" pod "f213cf23-67be-4699-828e-23a0a966c12a" (UID: "f213cf23-67be-4699-828e-23a0a966c12a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:49:11.353647 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.353478 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-workload-certs" pod "f213cf23-67be-4699-828e-23a0a966c12a" (UID: "f213cf23-67be-4699-828e-23a0a966c12a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:49:11.353877 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.353402 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f213cf23-67be-4699-828e-23a0a966c12a-kube-api-access-tlm89\" (UniqueName: \"kubernetes.io/projected/f213cf23-67be-4699-828e-23a0a966c12a-kube-api-access-tlm89\") pod \"f213cf23-67be-4699-828e-23a0a966c12a\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " Nov 03 15:49:11.353877 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.353812 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-workload-socket\") pod \"f213cf23-67be-4699-828e-23a0a966c12a\" (UID: \"f213cf23-67be-4699-828e-23a0a966c12a\") " Nov 03 15:49:11.353877 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.353855 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f213cf23-67be-4699-828e-23a0a966c12a-istiod-ca-cert" pod "f213cf23-67be-4699-828e-23a0a966c12a" (UID: "f213cf23-67be-4699-828e-23a0a966c12a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:49:11.354077 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.354057 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-workload-socket" pod "f213cf23-67be-4699-828e-23a0a966c12a" (UID: "f213cf23-67be-4699-828e-23a0a966c12a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:49:11.354315 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.354292 4290 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-istio-data\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.354445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.354317 4290 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-credential-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.354445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.354331 4290 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-workload-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.354445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.354344 4290 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f213cf23-67be-4699-828e-23a0a966c12a-istiod-ca-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.354445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.354355 4290 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-workload-socket\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.355768 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.355746 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f213cf23-67be-4699-828e-23a0a966c12a-istio-token" pod "f213cf23-67be-4699-828e-23a0a966c12a" (UID: "f213cf23-67be-4699-828e-23a0a966c12a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:49:11.355968 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.355939 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f213cf23-67be-4699-828e-23a0a966c12a-istio-podinfo" pod "f213cf23-67be-4699-828e-23a0a966c12a" (UID: "f213cf23-67be-4699-828e-23a0a966c12a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:49:11.356276 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.356257 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-istio-envoy" pod "f213cf23-67be-4699-828e-23a0a966c12a" (UID: "f213cf23-67be-4699-828e-23a0a966c12a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:49:11.356342 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.356278 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f213cf23-67be-4699-828e-23a0a966c12a-kube-api-access-tlm89" pod "f213cf23-67be-4699-828e-23a0a966c12a" (UID: "f213cf23-67be-4699-828e-23a0a966c12a"). InnerVolumeSpecName "kube-api-access-tlm89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:49:11.416289 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.416253 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f6786cb5-xfb8k"] Nov 03 15:49:11.416485 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.416464 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-65f6786cb5-xfb8k" podUID="7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23" containerName="manager" containerID="cri-o://3d1d56bc79b7cf0547e62814ff9316e672d19b8847d1c53e2301fdf6a0d1a01f" gracePeriod=30 Nov 03 15:49:11.455634 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.455597 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlm89\" (UniqueName: \"kubernetes.io/projected/f213cf23-67be-4699-828e-23a0a966c12a-kube-api-access-tlm89\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.455813 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.455651 4290 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f213cf23-67be-4699-828e-23a0a966c12a-istio-token\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.455813 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.455662 4290 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f213cf23-67be-4699-828e-23a0a966c12a-istio-podinfo\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.455813 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.455671 4290 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f213cf23-67be-4699-828e-23a0a966c12a-istio-envoy\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.611494 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.611419 4290 generic.go:356] "Generic (PLEG): container finished" podID="7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23" containerID="3d1d56bc79b7cf0547e62814ff9316e672d19b8847d1c53e2301fdf6a0d1a01f" exitCode=0 Nov 03 15:49:11.611665 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.611490 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f6786cb5-xfb8k" event={"ID":"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23","Type":"ContainerDied","Data":"3d1d56bc79b7cf0547e62814ff9316e672d19b8847d1c53e2301fdf6a0d1a01f"} Nov 03 15:49:11.613301 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.613275 4290 generic.go:356] "Generic (PLEG): container finished" podID="f213cf23-67be-4699-828e-23a0a966c12a" containerID="946748a29710d727cba118f7cda64bdc989bde662f7da4e72167a1b11fe0000a" exitCode=0 Nov 03 15:49:11.613428 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.613347 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" event={"ID":"f213cf23-67be-4699-828e-23a0a966c12a","Type":"ContainerDied","Data":"946748a29710d727cba118f7cda64bdc989bde662f7da4e72167a1b11fe0000a"} Nov 03 15:49:11.613428 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.613353 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" Nov 03 15:49:11.613428 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.613397 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl" event={"ID":"f213cf23-67be-4699-828e-23a0a966c12a","Type":"ContainerDied","Data":"4023dfd9c2aa3cb673305ff17e8caddb9d34cff53cc175290b08a98838ccfb2d"} Nov 03 15:49:11.613592 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.613428 4290 scope.go:122] "RemoveContainer" containerID="946748a29710d727cba118f7cda64bdc989bde662f7da4e72167a1b11fe0000a" Nov 03 15:49:11.626669 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.626647 4290 scope.go:122] "RemoveContainer" containerID="946748a29710d727cba118f7cda64bdc989bde662f7da4e72167a1b11fe0000a" Nov 03 15:49:11.627121 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:49:11.627085 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946748a29710d727cba118f7cda64bdc989bde662f7da4e72167a1b11fe0000a\": container with ID starting with 946748a29710d727cba118f7cda64bdc989bde662f7da4e72167a1b11fe0000a not found: ID does not exist" containerID="946748a29710d727cba118f7cda64bdc989bde662f7da4e72167a1b11fe0000a" Nov 03 15:49:11.627225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.627134 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946748a29710d727cba118f7cda64bdc989bde662f7da4e72167a1b11fe0000a"} err="failed to get container status \"946748a29710d727cba118f7cda64bdc989bde662f7da4e72167a1b11fe0000a\": rpc error: code = NotFound desc = could not find container \"946748a29710d727cba118f7cda64bdc989bde662f7da4e72167a1b11fe0000a\": container with ID starting with 946748a29710d727cba118f7cda64bdc989bde662f7da4e72167a1b11fe0000a not found: ID does not exist" Nov 03 15:49:11.666304 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.666282 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f6786cb5-xfb8k" Nov 03 15:49:11.758136 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.758106 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-kube-api-access-ngnx9\" (UniqueName: \"kubernetes.io/projected/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-kube-api-access-ngnx9\") pod \"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23\" (UID: \"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23\") " Nov 03 15:49:11.758333 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.758188 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-metrics-certs\") pod \"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23\" (UID: \"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23\") " Nov 03 15:49:11.758333 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.758244 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-webhook-cert\" (UniqueName: \"kubernetes.io/secret/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-webhook-cert\") pod \"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23\" (UID: \"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23\") " Nov 03 15:49:11.761049 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.761021 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-webhook-cert" pod "7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23" (UID: "7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:49:11.761210 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.761177 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-kube-api-access-ngnx9" pod "7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23" (UID: "7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23"). InnerVolumeSpecName "kube-api-access-ngnx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:49:11.761277 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.761242 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-metrics-certs" pod "7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23" (UID: "7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:49:11.859979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.859944 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngnx9\" (UniqueName: \"kubernetes.io/projected/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-kube-api-access-ngnx9\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.859979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.859974 4290 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-metrics-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.859979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.859983 4290 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23-webhook-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.913596 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.913563 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl"] Nov 03 15:49:11.920521 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:11.920493 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-7klfl"] Nov 03 15:49:12.622495 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:12.622462 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f6786cb5-xfb8k" Nov 03 15:49:12.623005 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:12.622504 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f6786cb5-xfb8k" event={"ID":"7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23","Type":"ContainerDied","Data":"e63ba32c308334b7ba1200bd43c713425a0ed029b25cb26d7a74126be7da61a0"} Nov 03 15:49:12.623005 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:12.622564 4290 scope.go:122] "RemoveContainer" containerID="3d1d56bc79b7cf0547e62814ff9316e672d19b8847d1c53e2301fdf6a0d1a01f" Nov 03 15:49:12.678365 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:12.678339 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f213cf23-67be-4699-828e-23a0a966c12a" path="/var/lib/kubelet/pods/f213cf23-67be-4699-828e-23a0a966c12a/volumes" Nov 03 15:49:12.791739 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:12.791691 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f6786cb5-xfb8k"] Nov 03 15:49:12.841476 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:12.841446 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65f6786cb5-xfb8k"] Nov 03 15:49:13.681261 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681233 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65f6786cb5-jgrll"] Nov 03 15:49:13.681710 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681676 4290 cpu_manager.go:414] "containerMap: removing container" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" Nov 03 15:49:13.681762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681711 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" Nov 03 15:49:13.681762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681732 4290 cpu_manager.go:414] "containerMap: removing container" podUID="0866b376-9269-41cf-aa1a-d29d433718e7" containerName="payload-pre-processing" Nov 03 15:49:13.681762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681738 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0866b376-9269-41cf-aa1a-d29d433718e7" containerName="payload-pre-processing" Nov 03 15:49:13.681762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681755 4290 cpu_manager.go:414] "containerMap: removing container" podUID="d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36" containerName="payload-processing" Nov 03 15:49:13.681762 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681760 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36" containerName="payload-processing" Nov 03 15:49:13.681926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681774 4290 cpu_manager.go:414] "containerMap: removing container" podUID="7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23" containerName="manager" Nov 03 15:49:13.681926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681778 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23" containerName="manager" Nov 03 15:49:13.681926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681789 4290 cpu_manager.go:414] "containerMap: removing container" podUID="5e47873e-a311-4b93-ac35-a1b02e509efd" containerName="revoke-keys" Nov 03 15:49:13.681926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681794 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e47873e-a311-4b93-ac35-a1b02e509efd" containerName="revoke-keys" Nov 03 15:49:13.681926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681803 4290 cpu_manager.go:414] "containerMap: removing container" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" Nov 03 15:49:13.681926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681808 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" Nov 03 15:49:13.681926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681869 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="0866b376-9269-41cf-aa1a-d29d433718e7" containerName="payload-pre-processing" Nov 03 15:49:13.681926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681878 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c7a85c-0cc1-4dcb-ae0d-45ea7a625c36" containerName="payload-processing" Nov 03 15:49:13.681926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681885 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23" containerName="manager" Nov 03 15:49:13.681926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681891 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e47873e-a311-4b93-ac35-a1b02e509efd" containerName="revoke-keys" Nov 03 15:49:13.681926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681897 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed595ae7-8d05-42d9-84e8-4a370338dcb8" containerName="main" Nov 03 15:49:13.681926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.681903 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="f213cf23-67be-4699-828e-23a0a966c12a" containerName="istio-proxy" Nov 03 15:49:13.685996 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.685979 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f6786cb5-jgrll" Nov 03 15:49:13.779843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.779812 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lffdp\" (UniqueName: \"kubernetes.io/projected/c3566ffb-589a-4799-ac55-237a843e5936-kube-api-access-lffdp\") pod \"maas-controller-65f6786cb5-jgrll\" (UID: \"c3566ffb-589a-4799-ac55-237a843e5936\") " pod="opendatahub/maas-controller-65f6786cb5-jgrll" Nov 03 15:49:13.780039 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.779878 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3566ffb-589a-4799-ac55-237a843e5936-webhook-cert\") pod \"maas-controller-65f6786cb5-jgrll\" (UID: \"c3566ffb-589a-4799-ac55-237a843e5936\") " pod="opendatahub/maas-controller-65f6786cb5-jgrll" Nov 03 15:49:13.780039 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.779982 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3566ffb-589a-4799-ac55-237a843e5936-metrics-certs\") pod \"maas-controller-65f6786cb5-jgrll\" (UID: \"c3566ffb-589a-4799-ac55-237a843e5936\") " pod="opendatahub/maas-controller-65f6786cb5-jgrll" Nov 03 15:49:13.790039 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.790012 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 15:49:13.790249 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.790181 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 15:49:13.790249 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.790205 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 15:49:13.790369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.790284 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zpmnn\"" Nov 03 15:49:13.793884 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.793859 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f6786cb5-jgrll"] Nov 03 15:49:13.880838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.880797 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3566ffb-589a-4799-ac55-237a843e5936-webhook-cert\") pod \"maas-controller-65f6786cb5-jgrll\" (UID: \"c3566ffb-589a-4799-ac55-237a843e5936\") " pod="opendatahub/maas-controller-65f6786cb5-jgrll" Nov 03 15:49:13.880838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.880848 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3566ffb-589a-4799-ac55-237a843e5936-metrics-certs\") pod \"maas-controller-65f6786cb5-jgrll\" (UID: \"c3566ffb-589a-4799-ac55-237a843e5936\") " pod="opendatahub/maas-controller-65f6786cb5-jgrll" Nov 03 15:49:13.881059 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.880928 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lffdp\" (UniqueName: \"kubernetes.io/projected/c3566ffb-589a-4799-ac55-237a843e5936-kube-api-access-lffdp\") pod \"maas-controller-65f6786cb5-jgrll\" (UID: \"c3566ffb-589a-4799-ac55-237a843e5936\") " pod="opendatahub/maas-controller-65f6786cb5-jgrll" Nov 03 15:49:13.883526 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.883500 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3566ffb-589a-4799-ac55-237a843e5936-webhook-cert\") pod \"maas-controller-65f6786cb5-jgrll\" (UID: \"c3566ffb-589a-4799-ac55-237a843e5936\") " pod="opendatahub/maas-controller-65f6786cb5-jgrll" Nov 03 15:49:13.883642 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.883567 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3566ffb-589a-4799-ac55-237a843e5936-metrics-certs\") pod \"maas-controller-65f6786cb5-jgrll\" (UID: \"c3566ffb-589a-4799-ac55-237a843e5936\") " pod="opendatahub/maas-controller-65f6786cb5-jgrll" Nov 03 15:49:13.945199 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.945128 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lffdp\" (UniqueName: \"kubernetes.io/projected/c3566ffb-589a-4799-ac55-237a843e5936-kube-api-access-lffdp\") pod \"maas-controller-65f6786cb5-jgrll\" (UID: \"c3566ffb-589a-4799-ac55-237a843e5936\") " pod="opendatahub/maas-controller-65f6786cb5-jgrll" Nov 03 15:49:13.999374 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:13.999342 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f6786cb5-jgrll" Nov 03 15:49:14.381977 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:14.381952 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f6786cb5-jgrll"] Nov 03 15:49:14.383070 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:49:14.383048 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3566ffb_589a_4799_ac55_237a843e5936.slice/crio-ac3e1b9beafa5f02458addcc0bb6dac16731b59669cb02f5454367845dc4fe1e WatchSource:0}: Error finding container ac3e1b9beafa5f02458addcc0bb6dac16731b59669cb02f5454367845dc4fe1e: Status 404 returned error can't find the container with id ac3e1b9beafa5f02458addcc0bb6dac16731b59669cb02f5454367845dc4fe1e Nov 03 15:49:14.384517 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:14.384499 4290 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:49:14.645105 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:14.645019 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f6786cb5-jgrll" event={"ID":"c3566ffb-589a-4799-ac55-237a843e5936","Type":"ContainerStarted","Data":"ac3e1b9beafa5f02458addcc0bb6dac16731b59669cb02f5454367845dc4fe1e"} Nov 03 15:49:14.677452 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:14.677421 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23" path="/var/lib/kubelet/pods/7cb8ddc9-3d11-42bb-870f-1d0ca13f8d23/volumes" Nov 03 15:49:15.656500 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:15.656465 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f6786cb5-jgrll" event={"ID":"c3566ffb-589a-4799-ac55-237a843e5936","Type":"ContainerStarted","Data":"cfbe812ad7c40f6dca8b87856881c7f57e6555c6cb17e469f2f46f6e9069ad3d"} Nov 03 15:49:15.656985 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:15.656526 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65f6786cb5-jgrll" Nov 03 15:49:15.778952 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:15.778892 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-65f6786cb5-jgrll" podStartSLOduration=1.824371894 podStartE2EDuration="2.778877604s" totalImagesPullingTime="954.50571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:49:15.775511671 +0000 UTC m=+2463.206469479" watchObservedRunningTime="2026-11-03 15:49:15.778877604 +0000 UTC m=+2463.209835409" Nov 03 15:49:17.762728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:17.762500 4290 scope.go:122] "RemoveContainer" containerID="da3ecf52f44466ca778d7ad1d0fa7a450de8184146404bc3c777831a7977860a" Nov 03 15:49:17.822324 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:17.822294 4290 scope.go:122] "RemoveContainer" containerID="4d64765ecc7d34a1049d0c418b4c6829d990ced5d985a04a79fb6bf063450ab9" Nov 03 15:49:17.831540 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:17.831515 4290 scope.go:122] "RemoveContainer" containerID="a50f03af0f077fa4ebedfb7b6eead41acd7178bb89127d127ceae747e50f62a6" Nov 03 15:49:17.843803 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:17.843481 4290 scope.go:122] "RemoveContainer" containerID="778d514436be429a02565ba9528908f6f162ad1c177ca9b21f1ac4d92365691a" Nov 03 15:49:26.718079 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:26.717997 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65f6786cb5-jgrll" Nov 03 15:49:37.193359 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.193322 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct"] Nov 03 15:49:37.199038 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.199018 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" Nov 03 15:49:37.273394 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.273359 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct"] Nov 03 15:49:37.301092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.301065 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:49:37.301454 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.301425 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:49:37.301532 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.301460 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:49:37.301532 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.301467 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:49:37.329485 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.329457 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9knjd\" (UniqueName: \"kubernetes.io/projected/49004735-0202-4237-be71-7bad208a6589-kube-api-access-9knjd\") pod \"maas-api-revoke-sub-85237e45-7t7ct\" (UID: \"49004735-0202-4237-be71-7bad208a6589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" Nov 03 15:49:37.329638 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.329525 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49004735-0202-4237-be71-7bad208a6589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-7t7ct\" (UID: \"49004735-0202-4237-be71-7bad208a6589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" Nov 03 15:49:37.329638 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.329604 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49004735-0202-4237-be71-7bad208a6589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-7t7ct\" (UID: \"49004735-0202-4237-be71-7bad208a6589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" Nov 03 15:49:37.430982 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.430947 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9knjd\" (UniqueName: \"kubernetes.io/projected/49004735-0202-4237-be71-7bad208a6589-kube-api-access-9knjd\") pod \"maas-api-revoke-sub-85237e45-7t7ct\" (UID: \"49004735-0202-4237-be71-7bad208a6589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" Nov 03 15:49:37.431184 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.431049 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49004735-0202-4237-be71-7bad208a6589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-7t7ct\" (UID: \"49004735-0202-4237-be71-7bad208a6589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" Nov 03 15:49:37.431184 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.431074 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49004735-0202-4237-be71-7bad208a6589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-7t7ct\" (UID: \"49004735-0202-4237-be71-7bad208a6589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" Nov 03 15:49:37.431838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.431814 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49004735-0202-4237-be71-7bad208a6589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-7t7ct\" (UID: \"49004735-0202-4237-be71-7bad208a6589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" Nov 03 15:49:37.433553 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.433531 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49004735-0202-4237-be71-7bad208a6589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-7t7ct\" (UID: \"49004735-0202-4237-be71-7bad208a6589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" Nov 03 15:49:37.493022 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.492947 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9knjd\" (UniqueName: \"kubernetes.io/projected/49004735-0202-4237-be71-7bad208a6589-kube-api-access-9knjd\") pod \"maas-api-revoke-sub-85237e45-7t7ct\" (UID: \"49004735-0202-4237-be71-7bad208a6589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" Nov 03 15:49:37.516084 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.516049 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" Nov 03 15:49:37.902693 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:37.902665 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct"] Nov 03 15:49:37.903181 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:49:37.903156 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49004735_0202_4237_be71_7bad208a6589.slice/crio-515c193d90c6ea94b7ca6ce61dab27e57484b5687db833c8c83038a98a69c414 WatchSource:0}: Error finding container 515c193d90c6ea94b7ca6ce61dab27e57484b5687db833c8c83038a98a69c414: Status 404 returned error can't find the container with id 515c193d90c6ea94b7ca6ce61dab27e57484b5687db833c8c83038a98a69c414 Nov 03 15:49:38.702101 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:49:38.702061 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:49:38.866894 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:38.866865 4290 generic.go:356] "Generic (PLEG): container finished" podID="49004735-0202-4237-be71-7bad208a6589" containerID="059fce8f8cca25743b2a4db18bed221a19ccbe9897a331889e46bd08cb8d6215" exitCode=0 Nov 03 15:49:38.867052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:38.866931 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" event={"ID":"49004735-0202-4237-be71-7bad208a6589","Type":"ContainerDied","Data":"059fce8f8cca25743b2a4db18bed221a19ccbe9897a331889e46bd08cb8d6215"} Nov 03 15:49:38.867052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:38.866979 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" event={"ID":"49004735-0202-4237-be71-7bad208a6589","Type":"ContainerStarted","Data":"515c193d90c6ea94b7ca6ce61dab27e57484b5687db833c8c83038a98a69c414"} Nov 03 15:49:40.012025 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:40.012002 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" Nov 03 15:49:40.159774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:40.159534 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49004735-0202-4237-be71-7bad208a6589-kube-api-access-9knjd\" (UniqueName: \"kubernetes.io/projected/49004735-0202-4237-be71-7bad208a6589-kube-api-access-9knjd\") pod \"49004735-0202-4237-be71-7bad208a6589\" (UID: \"49004735-0202-4237-be71-7bad208a6589\") " Nov 03 15:49:40.159774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:40.159613 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49004735-0202-4237-be71-7bad208a6589-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49004735-0202-4237-be71-7bad208a6589-maas-api-openshift-ca\") pod \"49004735-0202-4237-be71-7bad208a6589\" (UID: \"49004735-0202-4237-be71-7bad208a6589\") " Nov 03 15:49:40.160030 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:40.159837 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49004735-0202-4237-be71-7bad208a6589-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49004735-0202-4237-be71-7bad208a6589-maas-api-xks-ca\") pod \"49004735-0202-4237-be71-7bad208a6589\" (UID: \"49004735-0202-4237-be71-7bad208a6589\") " Nov 03 15:49:40.160653 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:40.160289 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49004735-0202-4237-be71-7bad208a6589-maas-api-openshift-ca" pod "49004735-0202-4237-be71-7bad208a6589" (UID: "49004735-0202-4237-be71-7bad208a6589"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:49:40.162479 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:40.162454 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49004735-0202-4237-be71-7bad208a6589-kube-api-access-9knjd" pod "49004735-0202-4237-be71-7bad208a6589" (UID: "49004735-0202-4237-be71-7bad208a6589"). InnerVolumeSpecName "kube-api-access-9knjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:49:40.163191 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:40.163165 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49004735-0202-4237-be71-7bad208a6589-maas-api-xks-ca" pod "49004735-0202-4237-be71-7bad208a6589" (UID: "49004735-0202-4237-be71-7bad208a6589"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:49:40.261105 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:40.261061 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49004735-0202-4237-be71-7bad208a6589-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:40.261105 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:40.261097 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9knjd\" (UniqueName: \"kubernetes.io/projected/49004735-0202-4237-be71-7bad208a6589-kube-api-access-9knjd\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:40.261105 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:40.261114 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49004735-0202-4237-be71-7bad208a6589-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:40.883439 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:40.883406 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" Nov 03 15:49:40.883439 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:40.883427 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct" event={"ID":"49004735-0202-4237-be71-7bad208a6589","Type":"ContainerDied","Data":"515c193d90c6ea94b7ca6ce61dab27e57484b5687db833c8c83038a98a69c414"} Nov 03 15:49:40.883663 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:40.883486 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="515c193d90c6ea94b7ca6ce61dab27e57484b5687db833c8c83038a98a69c414" Nov 03 15:49:49.900302 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:49.900268 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f6786cb5-jgrll"] Nov 03 15:49:49.900785 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:49.900460 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-65f6786cb5-jgrll" podUID="c3566ffb-589a-4799-ac55-237a843e5936" containerName="manager" containerID="cri-o://cfbe812ad7c40f6dca8b87856881c7f57e6555c6cb17e469f2f46f6e9069ad3d" gracePeriod=30 Nov 03 15:49:50.658317 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.658289 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f6786cb5-jgrll" Nov 03 15:49:50.761273 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.761242 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3566ffb-589a-4799-ac55-237a843e5936-webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3566ffb-589a-4799-ac55-237a843e5936-webhook-cert\") pod \"c3566ffb-589a-4799-ac55-237a843e5936\" (UID: \"c3566ffb-589a-4799-ac55-237a843e5936\") " Nov 03 15:49:50.761470 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.761414 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3566ffb-589a-4799-ac55-237a843e5936-kube-api-access-lffdp\" (UniqueName: \"kubernetes.io/projected/c3566ffb-589a-4799-ac55-237a843e5936-kube-api-access-lffdp\") pod \"c3566ffb-589a-4799-ac55-237a843e5936\" (UID: \"c3566ffb-589a-4799-ac55-237a843e5936\") " Nov 03 15:49:50.761470 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.761436 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3566ffb-589a-4799-ac55-237a843e5936-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3566ffb-589a-4799-ac55-237a843e5936-metrics-certs\") pod \"c3566ffb-589a-4799-ac55-237a843e5936\" (UID: \"c3566ffb-589a-4799-ac55-237a843e5936\") " Nov 03 15:49:50.763542 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.763505 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3566ffb-589a-4799-ac55-237a843e5936-webhook-cert" pod "c3566ffb-589a-4799-ac55-237a843e5936" (UID: "c3566ffb-589a-4799-ac55-237a843e5936"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:49:50.763651 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.763543 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3566ffb-589a-4799-ac55-237a843e5936-kube-api-access-lffdp" pod "c3566ffb-589a-4799-ac55-237a843e5936" (UID: "c3566ffb-589a-4799-ac55-237a843e5936"). InnerVolumeSpecName "kube-api-access-lffdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:49:50.763651 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.763578 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3566ffb-589a-4799-ac55-237a843e5936-metrics-certs" pod "c3566ffb-589a-4799-ac55-237a843e5936" (UID: "c3566ffb-589a-4799-ac55-237a843e5936"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:49:50.862809 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.862779 4290 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3566ffb-589a-4799-ac55-237a843e5936-webhook-cert\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:50.862809 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.862807 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lffdp\" (UniqueName: \"kubernetes.io/projected/c3566ffb-589a-4799-ac55-237a843e5936-kube-api-access-lffdp\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:50.862809 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.862818 4290 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3566ffb-589a-4799-ac55-237a843e5936-metrics-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:50.949747 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.949695 4290 generic.go:356] "Generic (PLEG): container finished" podID="c3566ffb-589a-4799-ac55-237a843e5936" containerID="cfbe812ad7c40f6dca8b87856881c7f57e6555c6cb17e469f2f46f6e9069ad3d" exitCode=0 Nov 03 15:49:50.950225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.949758 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f6786cb5-jgrll" Nov 03 15:49:50.950225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.949781 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f6786cb5-jgrll" event={"ID":"c3566ffb-589a-4799-ac55-237a843e5936","Type":"ContainerDied","Data":"cfbe812ad7c40f6dca8b87856881c7f57e6555c6cb17e469f2f46f6e9069ad3d"} Nov 03 15:49:50.950225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.949850 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f6786cb5-jgrll" event={"ID":"c3566ffb-589a-4799-ac55-237a843e5936","Type":"ContainerDied","Data":"ac3e1b9beafa5f02458addcc0bb6dac16731b59669cb02f5454367845dc4fe1e"} Nov 03 15:49:50.950225 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.949883 4290 scope.go:122] "RemoveContainer" containerID="cfbe812ad7c40f6dca8b87856881c7f57e6555c6cb17e469f2f46f6e9069ad3d" Nov 03 15:49:50.960939 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.960917 4290 scope.go:122] "RemoveContainer" containerID="cfbe812ad7c40f6dca8b87856881c7f57e6555c6cb17e469f2f46f6e9069ad3d" Nov 03 15:49:50.961203 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:49:50.961183 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfbe812ad7c40f6dca8b87856881c7f57e6555c6cb17e469f2f46f6e9069ad3d\": container with ID starting with cfbe812ad7c40f6dca8b87856881c7f57e6555c6cb17e469f2f46f6e9069ad3d not found: ID does not exist" containerID="cfbe812ad7c40f6dca8b87856881c7f57e6555c6cb17e469f2f46f6e9069ad3d" Nov 03 15:49:50.961263 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:50.961224 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfbe812ad7c40f6dca8b87856881c7f57e6555c6cb17e469f2f46f6e9069ad3d"} err="failed to get container status \"cfbe812ad7c40f6dca8b87856881c7f57e6555c6cb17e469f2f46f6e9069ad3d\": rpc error: code = NotFound desc = could not find container \"cfbe812ad7c40f6dca8b87856881c7f57e6555c6cb17e469f2f46f6e9069ad3d\": container with ID starting with cfbe812ad7c40f6dca8b87856881c7f57e6555c6cb17e469f2f46f6e9069ad3d not found: ID does not exist" Nov 03 15:49:51.117811 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:51.117733 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f6786cb5-jgrll"] Nov 03 15:49:51.168364 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:51.168333 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65f6786cb5-jgrll"] Nov 03 15:49:52.596367 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.596331 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65f6786cb5-5wwgn"] Nov 03 15:49:52.596846 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.596833 4290 cpu_manager.go:414] "containerMap: removing container" podUID="c3566ffb-589a-4799-ac55-237a843e5936" containerName="manager" Nov 03 15:49:52.596899 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.596847 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3566ffb-589a-4799-ac55-237a843e5936" containerName="manager" Nov 03 15:49:52.596899 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.596859 4290 cpu_manager.go:414] "containerMap: removing container" podUID="49004735-0202-4237-be71-7bad208a6589" containerName="revoke-keys" Nov 03 15:49:52.596899 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.596865 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49004735-0202-4237-be71-7bad208a6589" containerName="revoke-keys" Nov 03 15:49:52.597000 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.596929 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3566ffb-589a-4799-ac55-237a843e5936" containerName="manager" Nov 03 15:49:52.597000 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.596940 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="49004735-0202-4237-be71-7bad208a6589" containerName="revoke-keys" Nov 03 15:49:52.601092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.601075 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f6786cb5-5wwgn" Nov 03 15:49:52.686345 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.686306 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83bb0c49-dd3e-4323-b755-b78034bd8c30-webhook-cert\") pod \"maas-controller-65f6786cb5-5wwgn\" (UID: \"83bb0c49-dd3e-4323-b755-b78034bd8c30\") " pod="opendatahub/maas-controller-65f6786cb5-5wwgn" Nov 03 15:49:52.686545 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.686435 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83bb0c49-dd3e-4323-b755-b78034bd8c30-metrics-certs\") pod \"maas-controller-65f6786cb5-5wwgn\" (UID: \"83bb0c49-dd3e-4323-b755-b78034bd8c30\") " pod="opendatahub/maas-controller-65f6786cb5-5wwgn" Nov 03 15:49:52.686545 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.686487 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nswd6\" (UniqueName: \"kubernetes.io/projected/83bb0c49-dd3e-4323-b755-b78034bd8c30-kube-api-access-nswd6\") pod \"maas-controller-65f6786cb5-5wwgn\" (UID: \"83bb0c49-dd3e-4323-b755-b78034bd8c30\") " pod="opendatahub/maas-controller-65f6786cb5-5wwgn" Nov 03 15:49:52.687308 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.687284 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3566ffb-589a-4799-ac55-237a843e5936" path="/var/lib/kubelet/pods/c3566ffb-589a-4799-ac55-237a843e5936/volumes" Nov 03 15:49:52.703507 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.703475 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 15:49:52.703751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.703509 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 15:49:52.703751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.703522 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zpmnn\"" Nov 03 15:49:52.703903 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.703769 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 15:49:52.707480 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.707456 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f6786cb5-5wwgn"] Nov 03 15:49:52.788038 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.788002 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83bb0c49-dd3e-4323-b755-b78034bd8c30-webhook-cert\") pod \"maas-controller-65f6786cb5-5wwgn\" (UID: \"83bb0c49-dd3e-4323-b755-b78034bd8c30\") " pod="opendatahub/maas-controller-65f6786cb5-5wwgn" Nov 03 15:49:52.788245 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.788067 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83bb0c49-dd3e-4323-b755-b78034bd8c30-metrics-certs\") pod \"maas-controller-65f6786cb5-5wwgn\" (UID: \"83bb0c49-dd3e-4323-b755-b78034bd8c30\") " pod="opendatahub/maas-controller-65f6786cb5-5wwgn" Nov 03 15:49:52.788245 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.788102 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nswd6\" (UniqueName: \"kubernetes.io/projected/83bb0c49-dd3e-4323-b755-b78034bd8c30-kube-api-access-nswd6\") pod \"maas-controller-65f6786cb5-5wwgn\" (UID: \"83bb0c49-dd3e-4323-b755-b78034bd8c30\") " pod="opendatahub/maas-controller-65f6786cb5-5wwgn" Nov 03 15:49:52.790599 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.790572 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83bb0c49-dd3e-4323-b755-b78034bd8c30-metrics-certs\") pod \"maas-controller-65f6786cb5-5wwgn\" (UID: \"83bb0c49-dd3e-4323-b755-b78034bd8c30\") " pod="opendatahub/maas-controller-65f6786cb5-5wwgn" Nov 03 15:49:52.790721 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.790647 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83bb0c49-dd3e-4323-b755-b78034bd8c30-webhook-cert\") pod \"maas-controller-65f6786cb5-5wwgn\" (UID: \"83bb0c49-dd3e-4323-b755-b78034bd8c30\") " pod="opendatahub/maas-controller-65f6786cb5-5wwgn" Nov 03 15:49:52.849718 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.849632 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nswd6\" (UniqueName: \"kubernetes.io/projected/83bb0c49-dd3e-4323-b755-b78034bd8c30-kube-api-access-nswd6\") pod \"maas-controller-65f6786cb5-5wwgn\" (UID: \"83bb0c49-dd3e-4323-b755-b78034bd8c30\") " pod="opendatahub/maas-controller-65f6786cb5-5wwgn" Nov 03 15:49:52.915113 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:52.915074 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f6786cb5-5wwgn" Nov 03 15:49:53.302322 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:53.302290 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f6786cb5-5wwgn"] Nov 03 15:49:53.303302 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:49:53.303278 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bb0c49_dd3e_4323_b755_b78034bd8c30.slice/crio-74615ec942004851ea0d3b66787cbdc7319088d9ef4fa02701c05e92cfe879a0 WatchSource:0}: Error finding container 74615ec942004851ea0d3b66787cbdc7319088d9ef4fa02701c05e92cfe879a0: Status 404 returned error can't find the container with id 74615ec942004851ea0d3b66787cbdc7319088d9ef4fa02701c05e92cfe879a0 Nov 03 15:49:53.973055 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:53.973018 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f6786cb5-5wwgn" event={"ID":"83bb0c49-dd3e-4323-b755-b78034bd8c30","Type":"ContainerStarted","Data":"74615ec942004851ea0d3b66787cbdc7319088d9ef4fa02701c05e92cfe879a0"} Nov 03 15:49:54.980863 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:54.980821 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f6786cb5-5wwgn" event={"ID":"83bb0c49-dd3e-4323-b755-b78034bd8c30","Type":"ContainerStarted","Data":"fbe81e18d07748df031f7b8e1bf08037a1ecf34fd2693768c31c7da08873f5a2"} Nov 03 15:49:54.981311 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:54.981030 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65f6786cb5-5wwgn" Nov 03 15:49:55.101978 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:49:55.101917 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-65f6786cb5-5wwgn" podStartSLOduration=2.285845059 podStartE2EDuration="3.101898911s" totalImagesPullingTime="816.053852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:49:55.09917896 +0000 UTC m=+2502.530136762" watchObservedRunningTime="2026-11-03 15:49:55.101898911 +0000 UTC m=+2502.532856716" Nov 03 15:50:05.989726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:05.989669 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65f6786cb5-5wwgn" Nov 03 15:50:08.702977 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:50:08.702883 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:50:17.987161 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:17.987120 4290 scope.go:122] "RemoveContainer" containerID="fa89ea23c33191bea2165c6448206654fba39b5263f4e38f7bf13b24c5004132" Nov 03 15:50:38.707451 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:50:38.707410 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:50:40.556848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:40.556807 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pvhdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:50:40.557578 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:40.557542 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" podUID="c56971b8-db79-464a-bdaa-1a1e950f7c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:50:40.796928 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:40.796854 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-wl2z4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:50:40.797247 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:40.797218 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" podUID="a3cec29f-05df-404d-b346-53dda11adf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:50:46.833086 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:46.833047 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf"] Nov 03 15:50:46.847393 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:46.847365 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" Nov 03 15:50:46.902596 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:46.902563 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf"] Nov 03 15:50:46.947294 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:46.947261 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:50:46.947294 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:46.947284 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:50:46.947493 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:46.947331 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:50:46.949949 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:46.949929 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:50:46.996541 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:46.996515 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf0012a-424e-455e-86a5-66776eb99c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-k9tnf\" (UID: \"3cf0012a-424e-455e-86a5-66776eb99c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" Nov 03 15:50:46.996681 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:46.996569 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnl9v\" (UniqueName: \"kubernetes.io/projected/3cf0012a-424e-455e-86a5-66776eb99c93-kube-api-access-nnl9v\") pod \"maas-api-revoke-sub-c1e271a9-k9tnf\" (UID: \"3cf0012a-424e-455e-86a5-66776eb99c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" Nov 03 15:50:46.996769 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:46.996750 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf0012a-424e-455e-86a5-66776eb99c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-k9tnf\" (UID: \"3cf0012a-424e-455e-86a5-66776eb99c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" Nov 03 15:50:47.097816 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:47.097745 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnl9v\" (UniqueName: \"kubernetes.io/projected/3cf0012a-424e-455e-86a5-66776eb99c93-kube-api-access-nnl9v\") pod \"maas-api-revoke-sub-c1e271a9-k9tnf\" (UID: \"3cf0012a-424e-455e-86a5-66776eb99c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" Nov 03 15:50:47.097941 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:47.097886 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf0012a-424e-455e-86a5-66776eb99c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-k9tnf\" (UID: \"3cf0012a-424e-455e-86a5-66776eb99c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" Nov 03 15:50:47.097941 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:47.097936 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf0012a-424e-455e-86a5-66776eb99c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-k9tnf\" (UID: \"3cf0012a-424e-455e-86a5-66776eb99c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" Nov 03 15:50:47.098609 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:47.098589 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf0012a-424e-455e-86a5-66776eb99c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-k9tnf\" (UID: \"3cf0012a-424e-455e-86a5-66776eb99c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" Nov 03 15:50:47.100381 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:47.100346 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf0012a-424e-455e-86a5-66776eb99c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-k9tnf\" (UID: \"3cf0012a-424e-455e-86a5-66776eb99c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" Nov 03 15:50:47.159950 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:47.159920 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnl9v\" (UniqueName: \"kubernetes.io/projected/3cf0012a-424e-455e-86a5-66776eb99c93-kube-api-access-nnl9v\") pod \"maas-api-revoke-sub-c1e271a9-k9tnf\" (UID: \"3cf0012a-424e-455e-86a5-66776eb99c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" Nov 03 15:50:47.161717 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:47.161668 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" Nov 03 15:50:47.541252 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:47.541226 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf"] Nov 03 15:50:47.542577 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:50:47.542532 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf0012a_424e_455e_86a5_66776eb99c93.slice/crio-edf2723b92b15115f8478f01e7deadd1aa462b0c22d66a7897cd7b9431d07ac1 WatchSource:0}: Error finding container edf2723b92b15115f8478f01e7deadd1aa462b0c22d66a7897cd7b9431d07ac1: Status 404 returned error can't find the container with id edf2723b92b15115f8478f01e7deadd1aa462b0c22d66a7897cd7b9431d07ac1 Nov 03 15:50:48.364741 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:48.364683 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" event={"ID":"3cf0012a-424e-455e-86a5-66776eb99c93","Type":"ContainerStarted","Data":"ce8488e186103ab9fe25adb68d328eda373b7afa3bc67e38dfa8b67eccb703d3"} Nov 03 15:50:48.365226 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:48.364754 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" event={"ID":"3cf0012a-424e-455e-86a5-66776eb99c93","Type":"ContainerStarted","Data":"edf2723b92b15115f8478f01e7deadd1aa462b0c22d66a7897cd7b9431d07ac1"} Nov 03 15:50:48.481675 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:48.481607 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" podStartSLOduration=2.481589498 podStartE2EDuration="2.481589498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:50:48.480580546 +0000 UTC m=+2555.911538353" watchObservedRunningTime="2026-11-03 15:50:48.481589498 +0000 UTC m=+2555.912547306" Nov 03 15:50:49.373027 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:49.372991 4290 generic.go:356] "Generic (PLEG): container finished" podID="3cf0012a-424e-455e-86a5-66776eb99c93" containerID="ce8488e186103ab9fe25adb68d328eda373b7afa3bc67e38dfa8b67eccb703d3" exitCode=0 Nov 03 15:50:49.373457 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:49.373054 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" event={"ID":"3cf0012a-424e-455e-86a5-66776eb99c93","Type":"ContainerDied","Data":"ce8488e186103ab9fe25adb68d328eda373b7afa3bc67e38dfa8b67eccb703d3"} Nov 03 15:50:50.527596 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:50.527568 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" Nov 03 15:50:50.623889 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:50.623858 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf0012a-424e-455e-86a5-66776eb99c93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf0012a-424e-455e-86a5-66776eb99c93-maas-api-openshift-ca\") pod \"3cf0012a-424e-455e-86a5-66776eb99c93\" (UID: \"3cf0012a-424e-455e-86a5-66776eb99c93\") " Nov 03 15:50:50.624029 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:50.623971 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cf0012a-424e-455e-86a5-66776eb99c93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf0012a-424e-455e-86a5-66776eb99c93-maas-api-xks-ca\") pod \"3cf0012a-424e-455e-86a5-66776eb99c93\" (UID: \"3cf0012a-424e-455e-86a5-66776eb99c93\") " Nov 03 15:50:50.624194 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:50.624168 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf0012a-424e-455e-86a5-66776eb99c93-maas-api-openshift-ca" pod "3cf0012a-424e-455e-86a5-66776eb99c93" (UID: "3cf0012a-424e-455e-86a5-66776eb99c93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:50:50.625917 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:50.625892 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cf0012a-424e-455e-86a5-66776eb99c93-maas-api-xks-ca" pod "3cf0012a-424e-455e-86a5-66776eb99c93" (UID: "3cf0012a-424e-455e-86a5-66776eb99c93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:50:50.725052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:50.725021 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf0012a-424e-455e-86a5-66776eb99c93-kube-api-access-nnl9v\" (UniqueName: \"kubernetes.io/projected/3cf0012a-424e-455e-86a5-66776eb99c93-kube-api-access-nnl9v\") pod \"3cf0012a-424e-455e-86a5-66776eb99c93\" (UID: \"3cf0012a-424e-455e-86a5-66776eb99c93\") " Nov 03 15:50:50.725277 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:50.725262 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf0012a-424e-455e-86a5-66776eb99c93-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:50:50.725357 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:50.725278 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf0012a-424e-455e-86a5-66776eb99c93-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:50:50.727212 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:50.727188 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf0012a-424e-455e-86a5-66776eb99c93-kube-api-access-nnl9v" pod "3cf0012a-424e-455e-86a5-66776eb99c93" (UID: "3cf0012a-424e-455e-86a5-66776eb99c93"). InnerVolumeSpecName "kube-api-access-nnl9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:50:50.825805 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:50.825775 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnl9v\" (UniqueName: \"kubernetes.io/projected/3cf0012a-424e-455e-86a5-66776eb99c93-kube-api-access-nnl9v\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:50:51.387622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:51.387587 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" Nov 03 15:50:51.387836 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:51.387618 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf" event={"ID":"3cf0012a-424e-455e-86a5-66776eb99c93","Type":"ContainerDied","Data":"edf2723b92b15115f8478f01e7deadd1aa462b0c22d66a7897cd7b9431d07ac1"} Nov 03 15:50:51.387836 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:50:51.387665 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edf2723b92b15115f8478f01e7deadd1aa462b0c22d66a7897cd7b9431d07ac1" Nov 03 15:51:08.702969 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:51:08.702880 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:51:18.027628 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.027589 4290 scope.go:122] "RemoveContainer" containerID="43053fdafdebe78a27760834c5a65cabe96eadd852487e6236f1590049cd3557" Nov 03 15:51:18.039052 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.039025 4290 scope.go:122] "RemoveContainer" containerID="daaa9d3712c33845c3f7ccd1b2a9de00bd2ff68248005d78b72ad6ea4f81055e" Nov 03 15:51:18.049266 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.049240 4290 scope.go:122] "RemoveContainer" containerID="21c7f14d2cb2ad8161617d47ed6b3277c4b32514bd309852fc7598ecbb63927c" Nov 03 15:51:18.654325 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.654291 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6"] Nov 03 15:51:18.654835 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.654822 4290 cpu_manager.go:414] "containerMap: removing container" podUID="3cf0012a-424e-455e-86a5-66776eb99c93" containerName="revoke-keys" Nov 03 15:51:18.654887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.654837 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf0012a-424e-455e-86a5-66776eb99c93" containerName="revoke-keys" Nov 03 15:51:18.654923 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.654917 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf0012a-424e-455e-86a5-66776eb99c93" containerName="revoke-keys" Nov 03 15:51:18.659758 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.659735 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" Nov 03 15:51:18.719155 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.719115 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6"] Nov 03 15:51:18.724390 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.724341 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:51:18.739828 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.739791 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dpjm6\" (UID: \"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" Nov 03 15:51:18.740022 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.739846 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fnmf\" (UniqueName: \"kubernetes.io/projected/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-kube-api-access-5fnmf\") pod \"maas-api-revoke-sub-2e1f5b5f-dpjm6\" (UID: \"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" Nov 03 15:51:18.740072 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.740030 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dpjm6\" (UID: \"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" Nov 03 15:51:18.763558 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.763523 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:51:18.763774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.763532 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:51:18.763774 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.763582 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:51:18.840663 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.840621 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dpjm6\" (UID: \"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" Nov 03 15:51:18.840906 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.840736 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dpjm6\" (UID: \"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" Nov 03 15:51:18.840906 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.840775 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fnmf\" (UniqueName: \"kubernetes.io/projected/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-kube-api-access-5fnmf\") pod \"maas-api-revoke-sub-2e1f5b5f-dpjm6\" (UID: \"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" Nov 03 15:51:18.841448 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.841423 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dpjm6\" (UID: \"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" Nov 03 15:51:18.843221 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.843198 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dpjm6\" (UID: \"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" Nov 03 15:51:18.903205 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.903173 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fnmf\" (UniqueName: \"kubernetes.io/projected/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-kube-api-access-5fnmf\") pod \"maas-api-revoke-sub-2e1f5b5f-dpjm6\" (UID: \"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" Nov 03 15:51:18.973874 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:18.973782 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" Nov 03 15:51:19.357569 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:19.357516 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6"] Nov 03 15:51:19.358853 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:51:19.358827 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f31c536_5bd0_4e6e_a9e7_b0be825e3a8a.slice/crio-23c082b76cfc1849f16995130ba555fd5e4713b0cab1fb66dd5d3714e43701b0 WatchSource:0}: Error finding container 23c082b76cfc1849f16995130ba555fd5e4713b0cab1fb66dd5d3714e43701b0: Status 404 returned error can't find the container with id 23c082b76cfc1849f16995130ba555fd5e4713b0cab1fb66dd5d3714e43701b0 Nov 03 15:51:19.615591 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:19.615506 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" event={"ID":"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a","Type":"ContainerStarted","Data":"23c082b76cfc1849f16995130ba555fd5e4713b0cab1fb66dd5d3714e43701b0"} Nov 03 15:51:20.623296 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:20.623268 4290 generic.go:356] "Generic (PLEG): container finished" podID="8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a" containerID="ce4efb0924133640605172068b479b936d3a012010942794ebe6f0df95f3d667" exitCode=0 Nov 03 15:51:20.623751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:20.623350 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" event={"ID":"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a","Type":"ContainerDied","Data":"ce4efb0924133640605172068b479b936d3a012010942794ebe6f0df95f3d667"} Nov 03 15:51:21.778516 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:21.778483 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" Nov 03 15:51:21.872274 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:21.872237 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-maas-api-xks-ca\") pod \"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a\" (UID: \"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a\") " Nov 03 15:51:21.872467 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:21.872390 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-maas-api-openshift-ca\") pod \"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a\" (UID: \"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a\") " Nov 03 15:51:21.872467 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:21.872416 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-kube-api-access-5fnmf\" (UniqueName: \"kubernetes.io/projected/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-kube-api-access-5fnmf\") pod \"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a\" (UID: \"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a\") " Nov 03 15:51:21.872851 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:21.872824 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-maas-api-openshift-ca" pod "8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a" (UID: "8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:51:21.874526 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:21.874493 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-maas-api-xks-ca" pod "8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a" (UID: "8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:51:21.874716 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:21.874677 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-kube-api-access-5fnmf" pod "8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a" (UID: "8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a"). InnerVolumeSpecName "kube-api-access-5fnmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:51:21.974293 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:21.974250 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:51:21.974293 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:21.974285 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:51:21.974293 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:21.974297 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fnmf\" (UniqueName: \"kubernetes.io/projected/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a-kube-api-access-5fnmf\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:51:22.637994 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:22.637961 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" Nov 03 15:51:22.638199 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:22.637994 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6" event={"ID":"8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a","Type":"ContainerDied","Data":"23c082b76cfc1849f16995130ba555fd5e4713b0cab1fb66dd5d3714e43701b0"} Nov 03 15:51:22.638199 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:22.638043 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23c082b76cfc1849f16995130ba555fd5e4713b0cab1fb66dd5d3714e43701b0" Nov 03 15:51:31.152364 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.152325 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx"] Nov 03 15:51:31.152853 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.152838 4290 cpu_manager.go:414] "containerMap: removing container" podUID="8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a" containerName="revoke-keys" Nov 03 15:51:31.152903 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.152854 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a" containerName="revoke-keys" Nov 03 15:51:31.152963 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.152953 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a" containerName="revoke-keys" Nov 03 15:51:31.156429 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.156403 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" Nov 03 15:51:31.259925 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.259875 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:51:31.260116 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.260008 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:51:31.260116 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.260015 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:51:31.260116 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.260041 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:51:31.263530 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.263506 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx"] Nov 03 15:51:31.274057 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.274023 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkg76\" (UniqueName: \"kubernetes.io/projected/baa562a9-e6cd-4d77-9733-a1dd74490a3e-kube-api-access-vkg76\") pod \"maas-api-revoke-sub-ec843018-vkrxx\" (UID: \"baa562a9-e6cd-4d77-9733-a1dd74490a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" Nov 03 15:51:31.274221 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.274067 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa562a9-e6cd-4d77-9733-a1dd74490a3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-vkrxx\" (UID: \"baa562a9-e6cd-4d77-9733-a1dd74490a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" Nov 03 15:51:31.274277 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.274229 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa562a9-e6cd-4d77-9733-a1dd74490a3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-vkrxx\" (UID: \"baa562a9-e6cd-4d77-9733-a1dd74490a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" Nov 03 15:51:31.375463 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.375422 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa562a9-e6cd-4d77-9733-a1dd74490a3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-vkrxx\" (UID: \"baa562a9-e6cd-4d77-9733-a1dd74490a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" Nov 03 15:51:31.375680 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.375533 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkg76\" (UniqueName: \"kubernetes.io/projected/baa562a9-e6cd-4d77-9733-a1dd74490a3e-kube-api-access-vkg76\") pod \"maas-api-revoke-sub-ec843018-vkrxx\" (UID: \"baa562a9-e6cd-4d77-9733-a1dd74490a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" Nov 03 15:51:31.375680 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.375562 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa562a9-e6cd-4d77-9733-a1dd74490a3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-vkrxx\" (UID: \"baa562a9-e6cd-4d77-9733-a1dd74490a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" Nov 03 15:51:31.376337 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.376315 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa562a9-e6cd-4d77-9733-a1dd74490a3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-vkrxx\" (UID: \"baa562a9-e6cd-4d77-9733-a1dd74490a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" Nov 03 15:51:31.378092 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.378069 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa562a9-e6cd-4d77-9733-a1dd74490a3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-vkrxx\" (UID: \"baa562a9-e6cd-4d77-9733-a1dd74490a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" Nov 03 15:51:31.437594 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.437514 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkg76\" (UniqueName: \"kubernetes.io/projected/baa562a9-e6cd-4d77-9733-a1dd74490a3e-kube-api-access-vkg76\") pod \"maas-api-revoke-sub-ec843018-vkrxx\" (UID: \"baa562a9-e6cd-4d77-9733-a1dd74490a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" Nov 03 15:51:31.470676 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.470640 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" Nov 03 15:51:31.852590 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:31.852564 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx"] Nov 03 15:51:31.853995 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:51:31.853968 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa562a9_e6cd_4d77_9733_a1dd74490a3e.slice/crio-0ef66ce15d3a3fbb2ce020345de402dc698a4882b852d5c49eb13c832614c2e4 WatchSource:0}: Error finding container 0ef66ce15d3a3fbb2ce020345de402dc698a4882b852d5c49eb13c832614c2e4: Status 404 returned error can't find the container with id 0ef66ce15d3a3fbb2ce020345de402dc698a4882b852d5c49eb13c832614c2e4 Nov 03 15:51:32.715132 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:32.715093 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" event={"ID":"baa562a9-e6cd-4d77-9733-a1dd74490a3e","Type":"ContainerStarted","Data":"b05940e53e5689555c1013f6331ac066203a52a48ba8eabe4e0dfdf6e48100a4"} Nov 03 15:51:32.715551 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:32.715162 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" event={"ID":"baa562a9-e6cd-4d77-9733-a1dd74490a3e","Type":"ContainerStarted","Data":"0ef66ce15d3a3fbb2ce020345de402dc698a4882b852d5c49eb13c832614c2e4"} Nov 03 15:51:32.833377 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:32.833322 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" podStartSLOduration=1.833306371 podStartE2EDuration="1.833306371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:51:32.831761045 +0000 UTC m=+2600.262718851" watchObservedRunningTime="2026-11-03 15:51:32.833306371 +0000 UTC m=+2600.264264214" Nov 03 15:51:33.724328 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:33.724291 4290 generic.go:356] "Generic (PLEG): container finished" podID="baa562a9-e6cd-4d77-9733-a1dd74490a3e" containerID="b05940e53e5689555c1013f6331ac066203a52a48ba8eabe4e0dfdf6e48100a4" exitCode=0 Nov 03 15:51:33.724750 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:33.724382 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" event={"ID":"baa562a9-e6cd-4d77-9733-a1dd74490a3e","Type":"ContainerDied","Data":"b05940e53e5689555c1013f6331ac066203a52a48ba8eabe4e0dfdf6e48100a4"} Nov 03 15:51:34.875197 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:34.875164 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" Nov 03 15:51:35.012364 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:35.012280 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa562a9-e6cd-4d77-9733-a1dd74490a3e-kube-api-access-vkg76\" (UniqueName: \"kubernetes.io/projected/baa562a9-e6cd-4d77-9733-a1dd74490a3e-kube-api-access-vkg76\") pod \"baa562a9-e6cd-4d77-9733-a1dd74490a3e\" (UID: \"baa562a9-e6cd-4d77-9733-a1dd74490a3e\") " Nov 03 15:51:35.012532 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:35.012365 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baa562a9-e6cd-4d77-9733-a1dd74490a3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa562a9-e6cd-4d77-9733-a1dd74490a3e-maas-api-xks-ca\") pod \"baa562a9-e6cd-4d77-9733-a1dd74490a3e\" (UID: \"baa562a9-e6cd-4d77-9733-a1dd74490a3e\") " Nov 03 15:51:35.012532 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:35.012436 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa562a9-e6cd-4d77-9733-a1dd74490a3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa562a9-e6cd-4d77-9733-a1dd74490a3e-maas-api-openshift-ca\") pod \"baa562a9-e6cd-4d77-9733-a1dd74490a3e\" (UID: \"baa562a9-e6cd-4d77-9733-a1dd74490a3e\") " Nov 03 15:51:35.012818 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:35.012783 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa562a9-e6cd-4d77-9733-a1dd74490a3e-maas-api-openshift-ca" pod "baa562a9-e6cd-4d77-9733-a1dd74490a3e" (UID: "baa562a9-e6cd-4d77-9733-a1dd74490a3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:51:35.014773 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:35.014746 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa562a9-e6cd-4d77-9733-a1dd74490a3e-maas-api-xks-ca" pod "baa562a9-e6cd-4d77-9733-a1dd74490a3e" (UID: "baa562a9-e6cd-4d77-9733-a1dd74490a3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:51:35.014879 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:35.014838 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa562a9-e6cd-4d77-9733-a1dd74490a3e-kube-api-access-vkg76" pod "baa562a9-e6cd-4d77-9733-a1dd74490a3e" (UID: "baa562a9-e6cd-4d77-9733-a1dd74490a3e"). InnerVolumeSpecName "kube-api-access-vkg76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:51:35.051290 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:35.051251 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql"] Nov 03 15:51:35.054571 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:35.054540 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d86146d-znnql"] Nov 03 15:51:35.113434 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:35.113393 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkg76\" (UniqueName: \"kubernetes.io/projected/baa562a9-e6cd-4d77-9733-a1dd74490a3e-kube-api-access-vkg76\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:51:35.113434 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:35.113427 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa562a9-e6cd-4d77-9733-a1dd74490a3e-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:51:35.113434 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:35.113436 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa562a9-e6cd-4d77-9733-a1dd74490a3e-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:51:35.740991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:35.740953 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" event={"ID":"baa562a9-e6cd-4d77-9733-a1dd74490a3e","Type":"ContainerDied","Data":"0ef66ce15d3a3fbb2ce020345de402dc698a4882b852d5c49eb13c832614c2e4"} Nov 03 15:51:35.740991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:35.740981 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx" Nov 03 15:51:35.741216 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:35.741013 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ef66ce15d3a3fbb2ce020345de402dc698a4882b852d5c49eb13c832614c2e4" Nov 03 15:51:36.677054 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:36.677020 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67ad11f-ae4c-49ca-9763-d2eef45ac27d" path="/var/lib/kubelet/pods/d67ad11f-ae4c-49ca-9763-d2eef45ac27d/volumes" Nov 03 15:51:38.701660 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:51:38.701618 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:51:55.055584 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:55.055550 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n"] Nov 03 15:51:55.105036 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:55.105002 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859cf33f-vtp6n"] Nov 03 15:51:56.677891 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:51:56.677862 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05edca37-9fff-4f66-a2bd-a73c4ad46246" path="/var/lib/kubelet/pods/05edca37-9fff-4f66-a2bd-a73c4ad46246/volumes" Nov 03 15:52:06.433492 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.433458 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv"] Nov 03 15:52:06.433998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.433982 4290 cpu_manager.go:414] "containerMap: removing container" podUID="baa562a9-e6cd-4d77-9733-a1dd74490a3e" containerName="revoke-keys" Nov 03 15:52:06.434046 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.433999 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa562a9-e6cd-4d77-9733-a1dd74490a3e" containerName="revoke-keys" Nov 03 15:52:06.434086 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.434071 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa562a9-e6cd-4d77-9733-a1dd74490a3e" containerName="revoke-keys" Nov 03 15:52:06.437601 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.437584 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" Nov 03 15:52:06.475568 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.475534 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vmfvv\" (UID: \"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" Nov 03 15:52:06.475737 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.475647 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vmfvv\" (UID: \"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" Nov 03 15:52:06.475810 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.475749 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lsnt\" (UniqueName: \"kubernetes.io/projected/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-kube-api-access-2lsnt\") pod \"maas-api-revoke-sub-046e1343-vmfvv\" (UID: \"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" Nov 03 15:52:06.540825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.540789 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:52:06.541091 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.541071 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:52:06.541192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.541087 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:52:06.541192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.541088 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:52:06.544964 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.544940 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv"] Nov 03 15:52:06.576721 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.576676 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vmfvv\" (UID: \"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" Nov 03 15:52:06.576843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.576776 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vmfvv\" (UID: \"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" Nov 03 15:52:06.576843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.576839 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lsnt\" (UniqueName: \"kubernetes.io/projected/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-kube-api-access-2lsnt\") pod \"maas-api-revoke-sub-046e1343-vmfvv\" (UID: \"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" Nov 03 15:52:06.577452 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.577431 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vmfvv\" (UID: \"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" Nov 03 15:52:06.579235 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.579218 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vmfvv\" (UID: \"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" Nov 03 15:52:06.638428 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.638393 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lsnt\" (UniqueName: \"kubernetes.io/projected/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-kube-api-access-2lsnt\") pod \"maas-api-revoke-sub-046e1343-vmfvv\" (UID: \"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" Nov 03 15:52:06.750897 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:06.750821 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" Nov 03 15:52:07.049245 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:07.049161 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs"] Nov 03 15:52:07.052325 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:07.052303 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c405c767-2w5hs"] Nov 03 15:52:07.165748 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:07.164357 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv"] Nov 03 15:52:08.019079 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:08.019039 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" event={"ID":"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6","Type":"ContainerStarted","Data":"4f49d601fe89940cea93ee884dbf184e734789c3f207e145bca4ac5e337b51f1"} Nov 03 15:52:08.019486 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:08.019104 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" event={"ID":"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6","Type":"ContainerStarted","Data":"7e06796dd66305dfeddbf9eb0a25b0f249eb3c7131325bb69d6e056afaffa15c"} Nov 03 15:52:08.138611 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:08.138556 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" podStartSLOduration=2.138540167 podStartE2EDuration="2.138540167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:52:08.135727732 +0000 UTC m=+2635.566685535" watchObservedRunningTime="2026-11-03 15:52:08.138540167 +0000 UTC m=+2635.569497972" Nov 03 15:52:08.680069 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:08.679970 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14cc323b-4437-4c76-adf8-dc2a7dbd4da2" path="/var/lib/kubelet/pods/14cc323b-4437-4c76-adf8-dc2a7dbd4da2/volumes" Nov 03 15:52:08.701947 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:52:08.701916 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:52:09.027927 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:09.027840 4290 generic.go:356] "Generic (PLEG): container finished" podID="98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6" containerID="4f49d601fe89940cea93ee884dbf184e734789c3f207e145bca4ac5e337b51f1" exitCode=0 Nov 03 15:52:09.028307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:09.027930 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" event={"ID":"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6","Type":"ContainerDied","Data":"4f49d601fe89940cea93ee884dbf184e734789c3f207e145bca4ac5e337b51f1"} Nov 03 15:52:10.182622 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:10.182599 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" Nov 03 15:52:10.315022 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:10.314941 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-maas-api-xks-ca\") pod \"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6\" (UID: \"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6\") " Nov 03 15:52:10.315192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:10.315069 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-maas-api-openshift-ca\") pod \"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6\" (UID: \"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6\") " Nov 03 15:52:10.315192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:10.315171 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-kube-api-access-2lsnt\" (UniqueName: \"kubernetes.io/projected/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-kube-api-access-2lsnt\") pod \"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6\" (UID: \"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6\") " Nov 03 15:52:10.315518 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:10.315492 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-maas-api-openshift-ca" pod "98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6" (UID: "98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:52:10.315741 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:10.315680 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:52:10.317297 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:10.317263 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-maas-api-xks-ca" pod "98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6" (UID: "98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:52:10.317401 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:10.317357 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-kube-api-access-2lsnt" pod "98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6" (UID: "98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6"). InnerVolumeSpecName "kube-api-access-2lsnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:52:10.416455 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:10.416426 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:52:10.416455 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:10.416458 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lsnt\" (UniqueName: \"kubernetes.io/projected/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6-kube-api-access-2lsnt\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:52:11.043141 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:11.043067 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" Nov 03 15:52:11.043141 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:11.043082 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv" event={"ID":"98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6","Type":"ContainerDied","Data":"7e06796dd66305dfeddbf9eb0a25b0f249eb3c7131325bb69d6e056afaffa15c"} Nov 03 15:52:11.043141 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:11.043132 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e06796dd66305dfeddbf9eb0a25b0f249eb3c7131325bb69d6e056afaffa15c" Nov 03 15:52:17.214889 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.214850 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c"] Nov 03 15:52:17.215414 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.215398 4290 cpu_manager.go:414] "containerMap: removing container" podUID="98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6" containerName="revoke-keys" Nov 03 15:52:17.215484 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.215417 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6" containerName="revoke-keys" Nov 03 15:52:17.215547 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.215522 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6" containerName="revoke-keys" Nov 03 15:52:17.218920 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.218900 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" Nov 03 15:52:17.273576 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.273514 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:52:17.287176 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.287138 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4521bde-6c96-45a3-9f02-14129edc3c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-nzz4c\" (UID: \"d4521bde-6c96-45a3-9f02-14129edc3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" Nov 03 15:52:17.287383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.287197 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pvz4\" (UniqueName: \"kubernetes.io/projected/d4521bde-6c96-45a3-9f02-14129edc3c5d-kube-api-access-2pvz4\") pod \"maas-api-revoke-sub-d3fba056-nzz4c\" (UID: \"d4521bde-6c96-45a3-9f02-14129edc3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" Nov 03 15:52:17.287383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.287311 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4521bde-6c96-45a3-9f02-14129edc3c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-nzz4c\" (UID: \"d4521bde-6c96-45a3-9f02-14129edc3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" Nov 03 15:52:17.323305 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.323269 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:52:17.326752 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.326729 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c"] Nov 03 15:52:17.341317 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.341293 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:52:17.373339 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.373312 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:52:17.388897 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.388869 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4521bde-6c96-45a3-9f02-14129edc3c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-nzz4c\" (UID: \"d4521bde-6c96-45a3-9f02-14129edc3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" Nov 03 15:52:17.389013 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.388908 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pvz4\" (UniqueName: \"kubernetes.io/projected/d4521bde-6c96-45a3-9f02-14129edc3c5d-kube-api-access-2pvz4\") pod \"maas-api-revoke-sub-d3fba056-nzz4c\" (UID: \"d4521bde-6c96-45a3-9f02-14129edc3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" Nov 03 15:52:17.389013 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.388955 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4521bde-6c96-45a3-9f02-14129edc3c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-nzz4c\" (UID: \"d4521bde-6c96-45a3-9f02-14129edc3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" Nov 03 15:52:17.389505 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.389486 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4521bde-6c96-45a3-9f02-14129edc3c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-nzz4c\" (UID: \"d4521bde-6c96-45a3-9f02-14129edc3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" Nov 03 15:52:17.391443 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.391414 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4521bde-6c96-45a3-9f02-14129edc3c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-nzz4c\" (UID: \"d4521bde-6c96-45a3-9f02-14129edc3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" Nov 03 15:52:17.450993 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.450966 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pvz4\" (UniqueName: \"kubernetes.io/projected/d4521bde-6c96-45a3-9f02-14129edc3c5d-kube-api-access-2pvz4\") pod \"maas-api-revoke-sub-d3fba056-nzz4c\" (UID: \"d4521bde-6c96-45a3-9f02-14129edc3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" Nov 03 15:52:17.532787 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.532694 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" Nov 03 15:52:17.914509 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:17.914473 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c"] Nov 03 15:52:17.916101 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:52:17.916077 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4521bde_6c96_45a3_9f02_14129edc3c5d.slice/crio-ff9fece35c631139f9e6d7b2680d027fab36a10c0ff742f8ffa824dc5a6e96a8 WatchSource:0}: Error finding container ff9fece35c631139f9e6d7b2680d027fab36a10c0ff742f8ffa824dc5a6e96a8: Status 404 returned error can't find the container with id ff9fece35c631139f9e6d7b2680d027fab36a10c0ff742f8ffa824dc5a6e96a8 Nov 03 15:52:18.084990 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:18.084958 4290 scope.go:122] "RemoveContainer" containerID="f15fac05c3ad5562d9cb5af36e2adfe6760fbc561af7ae2613a54457a0b9c9ac" Nov 03 15:52:18.091820 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:18.091792 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" event={"ID":"d4521bde-6c96-45a3-9f02-14129edc3c5d","Type":"ContainerStarted","Data":"ff9fece35c631139f9e6d7b2680d027fab36a10c0ff742f8ffa824dc5a6e96a8"} Nov 03 15:52:18.096742 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:18.096720 4290 scope.go:122] "RemoveContainer" containerID="bdfc1dea020544ae5174ebdb4546169b618e02154fbe486bceae7958568d9350" Nov 03 15:52:19.103731 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:19.103629 4290 generic.go:356] "Generic (PLEG): container finished" podID="d4521bde-6c96-45a3-9f02-14129edc3c5d" containerID="c694fc34e0a0b5c40fb7a4f847f8022360eb3cc817647a798e54424029c35b46" exitCode=0 Nov 03 15:52:19.104133 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:19.103729 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" event={"ID":"d4521bde-6c96-45a3-9f02-14129edc3c5d","Type":"ContainerDied","Data":"c694fc34e0a0b5c40fb7a4f847f8022360eb3cc817647a798e54424029c35b46"} Nov 03 15:52:20.324678 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:20.324646 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" Nov 03 15:52:20.432880 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:20.432837 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4521bde-6c96-45a3-9f02-14129edc3c5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4521bde-6c96-45a3-9f02-14129edc3c5d-maas-api-xks-ca\") pod \"d4521bde-6c96-45a3-9f02-14129edc3c5d\" (UID: \"d4521bde-6c96-45a3-9f02-14129edc3c5d\") " Nov 03 15:52:20.433087 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:20.432900 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4521bde-6c96-45a3-9f02-14129edc3c5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4521bde-6c96-45a3-9f02-14129edc3c5d-maas-api-openshift-ca\") pod \"d4521bde-6c96-45a3-9f02-14129edc3c5d\" (UID: \"d4521bde-6c96-45a3-9f02-14129edc3c5d\") " Nov 03 15:52:20.433087 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:20.432985 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4521bde-6c96-45a3-9f02-14129edc3c5d-kube-api-access-2pvz4\" (UniqueName: \"kubernetes.io/projected/d4521bde-6c96-45a3-9f02-14129edc3c5d-kube-api-access-2pvz4\") pod \"d4521bde-6c96-45a3-9f02-14129edc3c5d\" (UID: \"d4521bde-6c96-45a3-9f02-14129edc3c5d\") " Nov 03 15:52:20.433386 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:20.433354 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4521bde-6c96-45a3-9f02-14129edc3c5d-maas-api-openshift-ca" pod "d4521bde-6c96-45a3-9f02-14129edc3c5d" (UID: "d4521bde-6c96-45a3-9f02-14129edc3c5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:52:20.435368 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:20.435341 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4521bde-6c96-45a3-9f02-14129edc3c5d-maas-api-xks-ca" pod "d4521bde-6c96-45a3-9f02-14129edc3c5d" (UID: "d4521bde-6c96-45a3-9f02-14129edc3c5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:52:20.435840 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:20.435818 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4521bde-6c96-45a3-9f02-14129edc3c5d-kube-api-access-2pvz4" pod "d4521bde-6c96-45a3-9f02-14129edc3c5d" (UID: "d4521bde-6c96-45a3-9f02-14129edc3c5d"). InnerVolumeSpecName "kube-api-access-2pvz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:52:20.534563 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:20.534509 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4521bde-6c96-45a3-9f02-14129edc3c5d-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:52:20.534563 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:20.534552 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4521bde-6c96-45a3-9f02-14129edc3c5d-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:52:20.534563 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:20.534570 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pvz4\" (UniqueName: \"kubernetes.io/projected/d4521bde-6c96-45a3-9f02-14129edc3c5d-kube-api-access-2pvz4\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:52:21.130813 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:21.130780 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" Nov 03 15:52:21.130813 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:21.130791 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c" event={"ID":"d4521bde-6c96-45a3-9f02-14129edc3c5d","Type":"ContainerDied","Data":"ff9fece35c631139f9e6d7b2680d027fab36a10c0ff742f8ffa824dc5a6e96a8"} Nov 03 15:52:21.131025 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:21.130869 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff9fece35c631139f9e6d7b2680d027fab36a10c0ff742f8ffa824dc5a6e96a8" Nov 03 15:52:38.749091 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:52:38.748966 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:52:55.205325 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.205290 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7"] Nov 03 15:52:55.206004 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.205980 4290 cpu_manager.go:414] "containerMap: removing container" podUID="d4521bde-6c96-45a3-9f02-14129edc3c5d" containerName="revoke-keys" Nov 03 15:52:55.206064 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.206010 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4521bde-6c96-45a3-9f02-14129edc3c5d" containerName="revoke-keys" Nov 03 15:52:55.206188 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.206173 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4521bde-6c96-45a3-9f02-14129edc3c5d" containerName="revoke-keys" Nov 03 15:52:55.209244 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.209226 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" Nov 03 15:52:55.282018 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.281983 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd533f31-7e0f-49a0-838a-312c84eeebb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-47xl7\" (UID: \"cd533f31-7e0f-49a0-838a-312c84eeebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" Nov 03 15:52:55.282204 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.282053 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f7lk\" (UniqueName: \"kubernetes.io/projected/cd533f31-7e0f-49a0-838a-312c84eeebb5-kube-api-access-6f7lk\") pod \"maas-api-revoke-sub-ae594979-47xl7\" (UID: \"cd533f31-7e0f-49a0-838a-312c84eeebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" Nov 03 15:52:55.282299 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.282270 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd533f31-7e0f-49a0-838a-312c84eeebb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-47xl7\" (UID: \"cd533f31-7e0f-49a0-838a-312c84eeebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" Nov 03 15:52:55.312824 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.312790 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:52:55.312985 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.312799 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:52:55.312985 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.312914 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:52:55.312985 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.312924 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:52:55.316403 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.316383 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7"] Nov 03 15:52:55.383254 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.383227 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd533f31-7e0f-49a0-838a-312c84eeebb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-47xl7\" (UID: \"cd533f31-7e0f-49a0-838a-312c84eeebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" Nov 03 15:52:55.383445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.383283 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd533f31-7e0f-49a0-838a-312c84eeebb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-47xl7\" (UID: \"cd533f31-7e0f-49a0-838a-312c84eeebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" Nov 03 15:52:55.383502 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.383485 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f7lk\" (UniqueName: \"kubernetes.io/projected/cd533f31-7e0f-49a0-838a-312c84eeebb5-kube-api-access-6f7lk\") pod \"maas-api-revoke-sub-ae594979-47xl7\" (UID: \"cd533f31-7e0f-49a0-838a-312c84eeebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" Nov 03 15:52:55.384169 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.384135 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd533f31-7e0f-49a0-838a-312c84eeebb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-47xl7\" (UID: \"cd533f31-7e0f-49a0-838a-312c84eeebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" Nov 03 15:52:55.385845 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.385821 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd533f31-7e0f-49a0-838a-312c84eeebb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-47xl7\" (UID: \"cd533f31-7e0f-49a0-838a-312c84eeebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" Nov 03 15:52:55.444688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.444659 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f7lk\" (UniqueName: \"kubernetes.io/projected/cd533f31-7e0f-49a0-838a-312c84eeebb5-kube-api-access-6f7lk\") pod \"maas-api-revoke-sub-ae594979-47xl7\" (UID: \"cd533f31-7e0f-49a0-838a-312c84eeebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" Nov 03 15:52:55.523144 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.523073 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" Nov 03 15:52:55.908361 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:55.908332 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7"] Nov 03 15:52:55.908742 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:52:55.908686 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd533f31_7e0f_49a0_838a_312c84eeebb5.slice/crio-3559f1be6ea8d0e00e9c233df30b4770deb0b377609f0f7b699d01a6eb62e508 WatchSource:0}: Error finding container 3559f1be6ea8d0e00e9c233df30b4770deb0b377609f0f7b699d01a6eb62e508: Status 404 returned error can't find the container with id 3559f1be6ea8d0e00e9c233df30b4770deb0b377609f0f7b699d01a6eb62e508 Nov 03 15:52:56.382026 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:56.381991 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" event={"ID":"cd533f31-7e0f-49a0-838a-312c84eeebb5","Type":"ContainerStarted","Data":"b7c04f39e63623cfe953d84bf74ebeea256aa7c008ea8be99cf4e383502f5c02"} Nov 03 15:52:56.382418 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:56.382045 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" event={"ID":"cd533f31-7e0f-49a0-838a-312c84eeebb5","Type":"ContainerStarted","Data":"3559f1be6ea8d0e00e9c233df30b4770deb0b377609f0f7b699d01a6eb62e508"} Nov 03 15:52:56.499398 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:56.499345 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" podStartSLOduration=1.499331571 podStartE2EDuration="1.499331571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:52:56.498679914 +0000 UTC m=+2683.929637718" watchObservedRunningTime="2026-11-03 15:52:56.499331571 +0000 UTC m=+2683.930289376" Nov 03 15:52:57.390274 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:57.390240 4290 generic.go:356] "Generic (PLEG): container finished" podID="cd533f31-7e0f-49a0-838a-312c84eeebb5" containerID="b7c04f39e63623cfe953d84bf74ebeea256aa7c008ea8be99cf4e383502f5c02" exitCode=0 Nov 03 15:52:57.390655 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:57.390284 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" event={"ID":"cd533f31-7e0f-49a0-838a-312c84eeebb5","Type":"ContainerDied","Data":"b7c04f39e63623cfe953d84bf74ebeea256aa7c008ea8be99cf4e383502f5c02"} Nov 03 15:52:58.534883 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:58.534858 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" Nov 03 15:52:58.617051 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:58.617020 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd533f31-7e0f-49a0-838a-312c84eeebb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd533f31-7e0f-49a0-838a-312c84eeebb5-maas-api-xks-ca\") pod \"cd533f31-7e0f-49a0-838a-312c84eeebb5\" (UID: \"cd533f31-7e0f-49a0-838a-312c84eeebb5\") " Nov 03 15:52:58.617221 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:58.617136 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd533f31-7e0f-49a0-838a-312c84eeebb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd533f31-7e0f-49a0-838a-312c84eeebb5-maas-api-openshift-ca\") pod \"cd533f31-7e0f-49a0-838a-312c84eeebb5\" (UID: \"cd533f31-7e0f-49a0-838a-312c84eeebb5\") " Nov 03 15:52:58.617221 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:58.617185 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd533f31-7e0f-49a0-838a-312c84eeebb5-kube-api-access-6f7lk\" (UniqueName: \"kubernetes.io/projected/cd533f31-7e0f-49a0-838a-312c84eeebb5-kube-api-access-6f7lk\") pod \"cd533f31-7e0f-49a0-838a-312c84eeebb5\" (UID: \"cd533f31-7e0f-49a0-838a-312c84eeebb5\") " Nov 03 15:52:58.617485 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:58.617463 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd533f31-7e0f-49a0-838a-312c84eeebb5-maas-api-openshift-ca" pod "cd533f31-7e0f-49a0-838a-312c84eeebb5" (UID: "cd533f31-7e0f-49a0-838a-312c84eeebb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:52:58.619233 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:58.619210 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd533f31-7e0f-49a0-838a-312c84eeebb5-maas-api-xks-ca" pod "cd533f31-7e0f-49a0-838a-312c84eeebb5" (UID: "cd533f31-7e0f-49a0-838a-312c84eeebb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:52:58.619352 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:58.619338 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd533f31-7e0f-49a0-838a-312c84eeebb5-kube-api-access-6f7lk" pod "cd533f31-7e0f-49a0-838a-312c84eeebb5" (UID: "cd533f31-7e0f-49a0-838a-312c84eeebb5"). InnerVolumeSpecName "kube-api-access-6f7lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:52:58.718707 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:58.718678 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f7lk\" (UniqueName: \"kubernetes.io/projected/cd533f31-7e0f-49a0-838a-312c84eeebb5-kube-api-access-6f7lk\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:52:58.718861 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:58.718722 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd533f31-7e0f-49a0-838a-312c84eeebb5-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:52:58.718861 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:58.718732 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd533f31-7e0f-49a0-838a-312c84eeebb5-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:52:59.404667 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:59.404631 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" Nov 03 15:52:59.404876 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:59.404677 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7" event={"ID":"cd533f31-7e0f-49a0-838a-312c84eeebb5","Type":"ContainerDied","Data":"3559f1be6ea8d0e00e9c233df30b4770deb0b377609f0f7b699d01a6eb62e508"} Nov 03 15:52:59.404876 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:52:59.404769 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3559f1be6ea8d0e00e9c233df30b4770deb0b377609f0f7b699d01a6eb62e508" Nov 03 15:53:08.707258 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:53:08.707172 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:53:13.055980 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:13.055940 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8"] Nov 03 15:53:13.103424 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:13.103386 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32cf43e3-wwcx8"] Nov 03 15:53:13.298465 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:13.298435 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:53:13.302114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:13.302084 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:53:13.308439 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:13.308352 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:53:13.311690 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:13.311668 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:53:14.695139 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:14.695102 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27d10df-774c-4fc9-a239-cf371d39c84d" path="/var/lib/kubelet/pods/e27d10df-774c-4fc9-a239-cf371d39c84d/volumes" Nov 03 15:53:18.132818 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:18.132780 4290 scope.go:122] "RemoveContainer" containerID="6a0213cbbbe4cb8a7d372da46b34fafa109cdb08ab13fbf106bfe760da2e934b" Nov 03 15:53:18.142610 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:18.142588 4290 scope.go:122] "RemoveContainer" containerID="9f2633dddf9ec8d07e3abd5dd8cf487b01c041bf892368d6c4ed26e342c6fcdd" Nov 03 15:53:18.151599 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:18.151580 4290 scope.go:122] "RemoveContainer" containerID="9b5bb1f39a0497e31390e3de4e8850592c0f2bbbf0ffd121e6b913a33e1cfea2" Nov 03 15:53:23.363292 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.363259 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm"] Nov 03 15:53:23.363875 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.363858 4290 cpu_manager.go:414] "containerMap: removing container" podUID="cd533f31-7e0f-49a0-838a-312c84eeebb5" containerName="revoke-keys" Nov 03 15:53:23.363946 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.363876 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd533f31-7e0f-49a0-838a-312c84eeebb5" containerName="revoke-keys" Nov 03 15:53:23.363998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.363988 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd533f31-7e0f-49a0-838a-312c84eeebb5" containerName="revoke-keys" Nov 03 15:53:23.368919 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.368898 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" Nov 03 15:53:23.401110 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.401083 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56e54f7-53c1-48f5-a568-fee939df5429-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kj8lm\" (UID: \"b56e54f7-53c1-48f5-a568-fee939df5429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" Nov 03 15:53:23.401252 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.401235 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56e54f7-53c1-48f5-a568-fee939df5429-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kj8lm\" (UID: \"b56e54f7-53c1-48f5-a568-fee939df5429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" Nov 03 15:53:23.401295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.401284 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfr84\" (UniqueName: \"kubernetes.io/projected/b56e54f7-53c1-48f5-a568-fee939df5429-kube-api-access-mfr84\") pod \"maas-api-revoke-sub-a4aef7d7-kj8lm\" (UID: \"b56e54f7-53c1-48f5-a568-fee939df5429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" Nov 03 15:53:23.425546 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.425511 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm"] Nov 03 15:53:23.469900 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.469872 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:53:23.470053 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.469913 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:53:23.470053 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.469931 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:53:23.471298 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.471277 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:53:23.502001 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.501969 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfr84\" (UniqueName: \"kubernetes.io/projected/b56e54f7-53c1-48f5-a568-fee939df5429-kube-api-access-mfr84\") pod \"maas-api-revoke-sub-a4aef7d7-kj8lm\" (UID: \"b56e54f7-53c1-48f5-a568-fee939df5429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" Nov 03 15:53:23.502168 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.502049 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56e54f7-53c1-48f5-a568-fee939df5429-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kj8lm\" (UID: \"b56e54f7-53c1-48f5-a568-fee939df5429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" Nov 03 15:53:23.502239 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.502187 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56e54f7-53c1-48f5-a568-fee939df5429-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kj8lm\" (UID: \"b56e54f7-53c1-48f5-a568-fee939df5429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" Nov 03 15:53:23.502901 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.502877 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56e54f7-53c1-48f5-a568-fee939df5429-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kj8lm\" (UID: \"b56e54f7-53c1-48f5-a568-fee939df5429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" Nov 03 15:53:23.504951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.504920 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56e54f7-53c1-48f5-a568-fee939df5429-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kj8lm\" (UID: \"b56e54f7-53c1-48f5-a568-fee939df5429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" Nov 03 15:53:23.563874 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.563846 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfr84\" (UniqueName: \"kubernetes.io/projected/b56e54f7-53c1-48f5-a568-fee939df5429-kube-api-access-mfr84\") pod \"maas-api-revoke-sub-a4aef7d7-kj8lm\" (UID: \"b56e54f7-53c1-48f5-a568-fee939df5429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" Nov 03 15:53:23.683777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:23.683747 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" Nov 03 15:53:24.067915 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:24.067886 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm"] Nov 03 15:53:24.070643 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:53:24.070612 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56e54f7_53c1_48f5_a568_fee939df5429.slice/crio-f15ab0435413d34e240f46153eb31a2ca43ce78752092d06ae4878d8f456e9a5 WatchSource:0}: Error finding container f15ab0435413d34e240f46153eb31a2ca43ce78752092d06ae4878d8f456e9a5: Status 404 returned error can't find the container with id f15ab0435413d34e240f46153eb31a2ca43ce78752092d06ae4878d8f456e9a5 Nov 03 15:53:24.601384 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:24.601294 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" event={"ID":"b56e54f7-53c1-48f5-a568-fee939df5429","Type":"ContainerStarted","Data":"d8e0177342e33d0f4d248ef39bb1dfe1f17cb958831ae70338977e83cb470e11"} Nov 03 15:53:24.601384 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:24.601343 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" event={"ID":"b56e54f7-53c1-48f5-a568-fee939df5429","Type":"ContainerStarted","Data":"f15ab0435413d34e240f46153eb31a2ca43ce78752092d06ae4878d8f456e9a5"} Nov 03 15:53:24.717862 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:24.717810 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" podStartSLOduration=1.717796661 podStartE2EDuration="1.717796661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:53:24.715762512 +0000 UTC m=+2712.146720319" watchObservedRunningTime="2026-11-03 15:53:24.717796661 +0000 UTC m=+2712.148754468" Nov 03 15:53:25.610425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:25.610395 4290 generic.go:356] "Generic (PLEG): container finished" podID="b56e54f7-53c1-48f5-a568-fee939df5429" containerID="d8e0177342e33d0f4d248ef39bb1dfe1f17cb958831ae70338977e83cb470e11" exitCode=0 Nov 03 15:53:25.610852 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:25.610448 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" event={"ID":"b56e54f7-53c1-48f5-a568-fee939df5429","Type":"ContainerDied","Data":"d8e0177342e33d0f4d248ef39bb1dfe1f17cb958831ae70338977e83cb470e11"} Nov 03 15:53:26.760597 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:26.760569 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" Nov 03 15:53:26.832503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:26.832474 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b56e54f7-53c1-48f5-a568-fee939df5429-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56e54f7-53c1-48f5-a568-fee939df5429-maas-api-xks-ca\") pod \"b56e54f7-53c1-48f5-a568-fee939df5429\" (UID: \"b56e54f7-53c1-48f5-a568-fee939df5429\") " Nov 03 15:53:26.832684 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:26.832589 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b56e54f7-53c1-48f5-a568-fee939df5429-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56e54f7-53c1-48f5-a568-fee939df5429-maas-api-openshift-ca\") pod \"b56e54f7-53c1-48f5-a568-fee939df5429\" (UID: \"b56e54f7-53c1-48f5-a568-fee939df5429\") " Nov 03 15:53:26.832684 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:26.832650 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56e54f7-53c1-48f5-a568-fee939df5429-kube-api-access-mfr84\" (UniqueName: \"kubernetes.io/projected/b56e54f7-53c1-48f5-a568-fee939df5429-kube-api-access-mfr84\") pod \"b56e54f7-53c1-48f5-a568-fee939df5429\" (UID: \"b56e54f7-53c1-48f5-a568-fee939df5429\") " Nov 03 15:53:26.832972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:26.832944 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56e54f7-53c1-48f5-a568-fee939df5429-maas-api-openshift-ca" pod "b56e54f7-53c1-48f5-a568-fee939df5429" (UID: "b56e54f7-53c1-48f5-a568-fee939df5429"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:53:26.834887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:26.834859 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56e54f7-53c1-48f5-a568-fee939df5429-kube-api-access-mfr84" pod "b56e54f7-53c1-48f5-a568-fee939df5429" (UID: "b56e54f7-53c1-48f5-a568-fee939df5429"). InnerVolumeSpecName "kube-api-access-mfr84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:53:26.834981 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:26.834937 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56e54f7-53c1-48f5-a568-fee939df5429-maas-api-xks-ca" pod "b56e54f7-53c1-48f5-a568-fee939df5429" (UID: "b56e54f7-53c1-48f5-a568-fee939df5429"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:53:26.934213 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:26.934179 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfr84\" (UniqueName: \"kubernetes.io/projected/b56e54f7-53c1-48f5-a568-fee939df5429-kube-api-access-mfr84\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:53:26.934213 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:26.934209 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56e54f7-53c1-48f5-a568-fee939df5429-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:53:26.934213 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:26.934218 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56e54f7-53c1-48f5-a568-fee939df5429-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:53:27.627347 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:27.627320 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" Nov 03 15:53:27.627524 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:27.627316 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm" event={"ID":"b56e54f7-53c1-48f5-a568-fee939df5429","Type":"ContainerDied","Data":"f15ab0435413d34e240f46153eb31a2ca43ce78752092d06ae4878d8f456e9a5"} Nov 03 15:53:27.627524 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:27.627449 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f15ab0435413d34e240f46153eb31a2ca43ce78752092d06ae4878d8f456e9a5" Nov 03 15:53:38.707203 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:53:38.707167 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:53:43.052838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:43.052808 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn"] Nov 03 15:53:43.101556 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:43.101521 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a257d6e-pvvdn"] Nov 03 15:53:44.701726 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:44.691989 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e47873e-a311-4b93-ac35-a1b02e509efd" path="/var/lib/kubelet/pods/5e47873e-a311-4b93-ac35-a1b02e509efd/volumes" Nov 03 15:53:54.134082 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.134042 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25"] Nov 03 15:53:54.134575 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.134560 4290 cpu_manager.go:414] "containerMap: removing container" podUID="b56e54f7-53c1-48f5-a568-fee939df5429" containerName="revoke-keys" Nov 03 15:53:54.134621 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.134577 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56e54f7-53c1-48f5-a568-fee939df5429" containerName="revoke-keys" Nov 03 15:53:54.134657 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.134653 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56e54f7-53c1-48f5-a568-fee939df5429" containerName="revoke-keys" Nov 03 15:53:54.137945 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.137929 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" Nov 03 15:53:54.193222 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.193191 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:53:54.215023 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.214996 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbx9h\" (UniqueName: \"kubernetes.io/projected/d35c1568-0a6d-4014-b4c3-a8695a69c325-kube-api-access-tbx9h\") pod \"maas-api-revoke-sub-79805ca9-28t25\" (UID: \"d35c1568-0a6d-4014-b4c3-a8695a69c325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" Nov 03 15:53:54.215181 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.215034 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35c1568-0a6d-4014-b4c3-a8695a69c325-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-28t25\" (UID: \"d35c1568-0a6d-4014-b4c3-a8695a69c325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" Nov 03 15:53:54.215181 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.215066 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35c1568-0a6d-4014-b4c3-a8695a69c325-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-28t25\" (UID: \"d35c1568-0a6d-4014-b4c3-a8695a69c325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" Nov 03 15:53:54.242626 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.242600 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:53:54.242772 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.242600 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:53:54.242772 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.242609 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:53:54.246086 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.246065 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25"] Nov 03 15:53:54.315961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.315931 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35c1568-0a6d-4014-b4c3-a8695a69c325-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-28t25\" (UID: \"d35c1568-0a6d-4014-b4c3-a8695a69c325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" Nov 03 15:53:54.316108 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.316090 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbx9h\" (UniqueName: \"kubernetes.io/projected/d35c1568-0a6d-4014-b4c3-a8695a69c325-kube-api-access-tbx9h\") pod \"maas-api-revoke-sub-79805ca9-28t25\" (UID: \"d35c1568-0a6d-4014-b4c3-a8695a69c325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" Nov 03 15:53:54.316152 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.316135 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35c1568-0a6d-4014-b4c3-a8695a69c325-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-28t25\" (UID: \"d35c1568-0a6d-4014-b4c3-a8695a69c325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" Nov 03 15:53:54.316841 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.316765 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35c1568-0a6d-4014-b4c3-a8695a69c325-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-28t25\" (UID: \"d35c1568-0a6d-4014-b4c3-a8695a69c325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" Nov 03 15:53:54.318516 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.318492 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35c1568-0a6d-4014-b4c3-a8695a69c325-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-28t25\" (UID: \"d35c1568-0a6d-4014-b4c3-a8695a69c325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" Nov 03 15:53:54.382073 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.382042 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbx9h\" (UniqueName: \"kubernetes.io/projected/d35c1568-0a6d-4014-b4c3-a8695a69c325-kube-api-access-tbx9h\") pod \"maas-api-revoke-sub-79805ca9-28t25\" (UID: \"d35c1568-0a6d-4014-b4c3-a8695a69c325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" Nov 03 15:53:54.451035 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.450963 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" Nov 03 15:53:54.831795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:54.831770 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25"] Nov 03 15:53:54.832999 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:53:54.832977 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35c1568_0a6d_4014_b4c3_a8695a69c325.slice/crio-565d2ccbf3ae9b97e679142e5252584d59e88c95acdc3b65b989c501f2549c09 WatchSource:0}: Error finding container 565d2ccbf3ae9b97e679142e5252584d59e88c95acdc3b65b989c501f2549c09: Status 404 returned error can't find the container with id 565d2ccbf3ae9b97e679142e5252584d59e88c95acdc3b65b989c501f2549c09 Nov 03 15:53:55.845168 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:55.845134 4290 generic.go:356] "Generic (PLEG): container finished" podID="d35c1568-0a6d-4014-b4c3-a8695a69c325" containerID="00d08d019c3341b4929e109dff837c16d57e3a2121b5d203645b60b54b47358b" exitCode=0 Nov 03 15:53:55.845556 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:55.845184 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" event={"ID":"d35c1568-0a6d-4014-b4c3-a8695a69c325","Type":"ContainerDied","Data":"00d08d019c3341b4929e109dff837c16d57e3a2121b5d203645b60b54b47358b"} Nov 03 15:53:55.845556 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:55.845255 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" event={"ID":"d35c1568-0a6d-4014-b4c3-a8695a69c325","Type":"ContainerStarted","Data":"565d2ccbf3ae9b97e679142e5252584d59e88c95acdc3b65b989c501f2549c09"} Nov 03 15:53:56.985925 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:56.985900 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" Nov 03 15:53:57.153935 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:57.153904 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35c1568-0a6d-4014-b4c3-a8695a69c325-kube-api-access-tbx9h\" (UniqueName: \"kubernetes.io/projected/d35c1568-0a6d-4014-b4c3-a8695a69c325-kube-api-access-tbx9h\") pod \"d35c1568-0a6d-4014-b4c3-a8695a69c325\" (UID: \"d35c1568-0a6d-4014-b4c3-a8695a69c325\") " Nov 03 15:53:57.154112 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:57.153990 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35c1568-0a6d-4014-b4c3-a8695a69c325-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35c1568-0a6d-4014-b4c3-a8695a69c325-maas-api-openshift-ca\") pod \"d35c1568-0a6d-4014-b4c3-a8695a69c325\" (UID: \"d35c1568-0a6d-4014-b4c3-a8695a69c325\") " Nov 03 15:53:57.154112 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:57.154060 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d35c1568-0a6d-4014-b4c3-a8695a69c325-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35c1568-0a6d-4014-b4c3-a8695a69c325-maas-api-xks-ca\") pod \"d35c1568-0a6d-4014-b4c3-a8695a69c325\" (UID: \"d35c1568-0a6d-4014-b4c3-a8695a69c325\") " Nov 03 15:53:57.154328 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:57.154304 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35c1568-0a6d-4014-b4c3-a8695a69c325-maas-api-openshift-ca" pod "d35c1568-0a6d-4014-b4c3-a8695a69c325" (UID: "d35c1568-0a6d-4014-b4c3-a8695a69c325"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:53:57.156171 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:57.156147 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d35c1568-0a6d-4014-b4c3-a8695a69c325-maas-api-xks-ca" pod "d35c1568-0a6d-4014-b4c3-a8695a69c325" (UID: "d35c1568-0a6d-4014-b4c3-a8695a69c325"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:53:57.156256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:57.156198 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35c1568-0a6d-4014-b4c3-a8695a69c325-kube-api-access-tbx9h" pod "d35c1568-0a6d-4014-b4c3-a8695a69c325" (UID: "d35c1568-0a6d-4014-b4c3-a8695a69c325"). InnerVolumeSpecName "kube-api-access-tbx9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:53:57.255267 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:57.255236 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbx9h\" (UniqueName: \"kubernetes.io/projected/d35c1568-0a6d-4014-b4c3-a8695a69c325-kube-api-access-tbx9h\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:53:57.255267 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:57.255264 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35c1568-0a6d-4014-b4c3-a8695a69c325-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:53:57.255462 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:57.255276 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35c1568-0a6d-4014-b4c3-a8695a69c325-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:53:57.859350 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:57.859321 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" Nov 03 15:53:57.859350 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:57.859346 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25" event={"ID":"d35c1568-0a6d-4014-b4c3-a8695a69c325","Type":"ContainerDied","Data":"565d2ccbf3ae9b97e679142e5252584d59e88c95acdc3b65b989c501f2549c09"} Nov 03 15:53:57.859566 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:53:57.859394 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="565d2ccbf3ae9b97e679142e5252584d59e88c95acdc3b65b989c501f2549c09" Nov 03 15:54:08.702149 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:54:08.702064 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:54:18.012678 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.012641 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf"] Nov 03 15:54:18.013190 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.013158 4290 cpu_manager.go:414] "containerMap: removing container" podUID="d35c1568-0a6d-4014-b4c3-a8695a69c325" containerName="revoke-keys" Nov 03 15:54:18.013190 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.013171 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35c1568-0a6d-4014-b4c3-a8695a69c325" containerName="revoke-keys" Nov 03 15:54:18.013274 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.013243 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35c1568-0a6d-4014-b4c3-a8695a69c325" containerName="revoke-keys" Nov 03 15:54:18.016498 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.016477 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" Nov 03 15:54:18.070425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.070396 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:54:18.081313 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.081274 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2ngxf\" (UID: \"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" Nov 03 15:54:18.081498 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.081333 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2ngxf\" (UID: \"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" Nov 03 15:54:18.081498 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.081429 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl78m\" (UniqueName: \"kubernetes.io/projected/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-kube-api-access-gl78m\") pod \"maas-api-revoke-sub-0b7e8edb-2ngxf\" (UID: \"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" Nov 03 15:54:18.120034 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.119997 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:54:18.120034 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.120010 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:54:18.120248 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.120079 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:54:18.123540 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.123522 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf"] Nov 03 15:54:18.182384 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.182349 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2ngxf\" (UID: \"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" Nov 03 15:54:18.182561 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.182392 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2ngxf\" (UID: \"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" Nov 03 15:54:18.182561 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.182499 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl78m\" (UniqueName: \"kubernetes.io/projected/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-kube-api-access-gl78m\") pod \"maas-api-revoke-sub-0b7e8edb-2ngxf\" (UID: \"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" Nov 03 15:54:18.183088 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.183065 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2ngxf\" (UID: \"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" Nov 03 15:54:18.184966 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.184942 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2ngxf\" (UID: \"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" Nov 03 15:54:18.185321 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.185302 4290 scope.go:122] "RemoveContainer" containerID="a24b974848c7cd62706b0dd0a193abc79aeef2ad5628152d264271ea0c395ca4" Nov 03 15:54:18.244549 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.244524 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl78m\" (UniqueName: \"kubernetes.io/projected/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-kube-api-access-gl78m\") pod \"maas-api-revoke-sub-0b7e8edb-2ngxf\" (UID: \"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" Nov 03 15:54:18.330986 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.330924 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" Nov 03 15:54:18.712012 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:18.711982 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf"] Nov 03 15:54:18.712845 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:54:18.712818 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1edee5_bb39_4ce5_8e4e_afeab678bfdf.slice/crio-92b1a1289cf7b231564038fdc22cdca2555e01e9be25d87e2f9c6eb855fe8afd WatchSource:0}: Error finding container 92b1a1289cf7b231564038fdc22cdca2555e01e9be25d87e2f9c6eb855fe8afd: Status 404 returned error can't find the container with id 92b1a1289cf7b231564038fdc22cdca2555e01e9be25d87e2f9c6eb855fe8afd Nov 03 15:54:19.055579 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:19.055551 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" event={"ID":"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf","Type":"ContainerStarted","Data":"92b1a1289cf7b231564038fdc22cdca2555e01e9be25d87e2f9c6eb855fe8afd"} Nov 03 15:54:20.069498 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:20.069461 4290 generic.go:356] "Generic (PLEG): container finished" podID="ac1edee5-bb39-4ce5-8e4e-afeab678bfdf" containerID="91c1a0d1b3ec2b9ca513a303a62d3a0ef0f479ff28d486675eb599a34b785fae" exitCode=0 Nov 03 15:54:20.069991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:20.069555 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" event={"ID":"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf","Type":"ContainerDied","Data":"91c1a0d1b3ec2b9ca513a303a62d3a0ef0f479ff28d486675eb599a34b785fae"} Nov 03 15:54:21.221564 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:21.221538 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" Nov 03 15:54:21.313900 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:21.313869 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-maas-api-openshift-ca\") pod \"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf\" (UID: \"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf\") " Nov 03 15:54:21.314089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:21.314030 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-kube-api-access-gl78m\" (UniqueName: \"kubernetes.io/projected/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-kube-api-access-gl78m\") pod \"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf\" (UID: \"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf\") " Nov 03 15:54:21.314089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:21.314076 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-maas-api-xks-ca\") pod \"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf\" (UID: \"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf\") " Nov 03 15:54:21.314251 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:21.314226 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-maas-api-openshift-ca" pod "ac1edee5-bb39-4ce5-8e4e-afeab678bfdf" (UID: "ac1edee5-bb39-4ce5-8e4e-afeab678bfdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:54:21.314395 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:21.314368 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:54:21.316291 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:21.316265 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-maas-api-xks-ca" pod "ac1edee5-bb39-4ce5-8e4e-afeab678bfdf" (UID: "ac1edee5-bb39-4ce5-8e4e-afeab678bfdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:54:21.316377 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:21.316314 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-kube-api-access-gl78m" pod "ac1edee5-bb39-4ce5-8e4e-afeab678bfdf" (UID: "ac1edee5-bb39-4ce5-8e4e-afeab678bfdf"). InnerVolumeSpecName "kube-api-access-gl78m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:54:21.415286 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:21.415251 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl78m\" (UniqueName: \"kubernetes.io/projected/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-kube-api-access-gl78m\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:54:21.415286 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:21.415281 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:54:22.082964 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:22.082933 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" Nov 03 15:54:22.083143 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:22.082963 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf" event={"ID":"ac1edee5-bb39-4ce5-8e4e-afeab678bfdf","Type":"ContainerDied","Data":"92b1a1289cf7b231564038fdc22cdca2555e01e9be25d87e2f9c6eb855fe8afd"} Nov 03 15:54:22.083143 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:22.083016 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92b1a1289cf7b231564038fdc22cdca2555e01e9be25d87e2f9c6eb855fe8afd" Nov 03 15:54:29.385218 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.385186 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62"] Nov 03 15:54:29.385934 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.385913 4290 cpu_manager.go:414] "containerMap: removing container" podUID="ac1edee5-bb39-4ce5-8e4e-afeab678bfdf" containerName="revoke-keys" Nov 03 15:54:29.386059 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.385938 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1edee5-bb39-4ce5-8e4e-afeab678bfdf" containerName="revoke-keys" Nov 03 15:54:29.386059 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.386046 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1edee5-bb39-4ce5-8e4e-afeab678bfdf" containerName="revoke-keys" Nov 03 15:54:29.389568 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.389512 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" Nov 03 15:54:29.489798 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.489763 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893e7840-ce60-4a01-9120-5531949b667f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fhw62\" (UID: \"893e7840-ce60-4a01-9120-5531949b667f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" Nov 03 15:54:29.489956 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.489828 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r58pz\" (UniqueName: \"kubernetes.io/projected/893e7840-ce60-4a01-9120-5531949b667f-kube-api-access-r58pz\") pod \"maas-api-revoke-sub-3d4156f1-fhw62\" (UID: \"893e7840-ce60-4a01-9120-5531949b667f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" Nov 03 15:54:29.489956 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.489912 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893e7840-ce60-4a01-9120-5531949b667f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fhw62\" (UID: \"893e7840-ce60-4a01-9120-5531949b667f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" Nov 03 15:54:29.492777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.492747 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:54:29.492777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.492762 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:54:29.492996 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.492756 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:54:29.492996 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.492897 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:54:29.496339 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.496320 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62"] Nov 03 15:54:29.591066 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.591030 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893e7840-ce60-4a01-9120-5531949b667f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fhw62\" (UID: \"893e7840-ce60-4a01-9120-5531949b667f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" Nov 03 15:54:29.591220 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.591096 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r58pz\" (UniqueName: \"kubernetes.io/projected/893e7840-ce60-4a01-9120-5531949b667f-kube-api-access-r58pz\") pod \"maas-api-revoke-sub-3d4156f1-fhw62\" (UID: \"893e7840-ce60-4a01-9120-5531949b667f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" Nov 03 15:54:29.591220 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.591127 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893e7840-ce60-4a01-9120-5531949b667f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fhw62\" (UID: \"893e7840-ce60-4a01-9120-5531949b667f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" Nov 03 15:54:29.591799 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.591775 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893e7840-ce60-4a01-9120-5531949b667f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fhw62\" (UID: \"893e7840-ce60-4a01-9120-5531949b667f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" Nov 03 15:54:29.593624 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.593598 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893e7840-ce60-4a01-9120-5531949b667f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fhw62\" (UID: \"893e7840-ce60-4a01-9120-5531949b667f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" Nov 03 15:54:29.653093 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.653026 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r58pz\" (UniqueName: \"kubernetes.io/projected/893e7840-ce60-4a01-9120-5531949b667f-kube-api-access-r58pz\") pod \"maas-api-revoke-sub-3d4156f1-fhw62\" (UID: \"893e7840-ce60-4a01-9120-5531949b667f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" Nov 03 15:54:29.703084 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:29.703049 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" Nov 03 15:54:30.086504 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:30.086478 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62"] Nov 03 15:54:30.087679 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:54:30.087652 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893e7840_ce60_4a01_9120_5531949b667f.slice/crio-29e540bcde2d0c563a67cf92a7e0e5d19b0a8f9767498607c337d39f0558cf6c WatchSource:0}: Error finding container 29e540bcde2d0c563a67cf92a7e0e5d19b0a8f9767498607c337d39f0558cf6c: Status 404 returned error can't find the container with id 29e540bcde2d0c563a67cf92a7e0e5d19b0a8f9767498607c337d39f0558cf6c Nov 03 15:54:30.140589 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:30.140556 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" event={"ID":"893e7840-ce60-4a01-9120-5531949b667f","Type":"ContainerStarted","Data":"29e540bcde2d0c563a67cf92a7e0e5d19b0a8f9767498607c337d39f0558cf6c"} Nov 03 15:54:31.148512 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:31.148480 4290 generic.go:356] "Generic (PLEG): container finished" podID="893e7840-ce60-4a01-9120-5531949b667f" containerID="4cd3a2f43d99ff3426ad96fe4e9c8e8a3112d7cd591acb47d175c7e346a8491f" exitCode=0 Nov 03 15:54:31.148926 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:31.148557 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" event={"ID":"893e7840-ce60-4a01-9120-5531949b667f","Type":"ContainerDied","Data":"4cd3a2f43d99ff3426ad96fe4e9c8e8a3112d7cd591acb47d175c7e346a8491f"} Nov 03 15:54:32.556734 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:32.554840 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" Nov 03 15:54:32.628280 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:32.628239 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/893e7840-ce60-4a01-9120-5531949b667f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893e7840-ce60-4a01-9120-5531949b667f-maas-api-xks-ca\") pod \"893e7840-ce60-4a01-9120-5531949b667f\" (UID: \"893e7840-ce60-4a01-9120-5531949b667f\") " Nov 03 15:54:32.628497 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:32.628413 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893e7840-ce60-4a01-9120-5531949b667f-kube-api-access-r58pz\" (UniqueName: \"kubernetes.io/projected/893e7840-ce60-4a01-9120-5531949b667f-kube-api-access-r58pz\") pod \"893e7840-ce60-4a01-9120-5531949b667f\" (UID: \"893e7840-ce60-4a01-9120-5531949b667f\") " Nov 03 15:54:32.628837 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:32.628812 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/893e7840-ce60-4a01-9120-5531949b667f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893e7840-ce60-4a01-9120-5531949b667f-maas-api-openshift-ca\") pod \"893e7840-ce60-4a01-9120-5531949b667f\" (UID: \"893e7840-ce60-4a01-9120-5531949b667f\") " Nov 03 15:54:32.629446 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:32.629328 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893e7840-ce60-4a01-9120-5531949b667f-maas-api-openshift-ca" pod "893e7840-ce60-4a01-9120-5531949b667f" (UID: "893e7840-ce60-4a01-9120-5531949b667f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:54:32.629597 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:32.629572 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893e7840-ce60-4a01-9120-5531949b667f-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:54:32.630778 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:32.630751 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893e7840-ce60-4a01-9120-5531949b667f-maas-api-xks-ca" pod "893e7840-ce60-4a01-9120-5531949b667f" (UID: "893e7840-ce60-4a01-9120-5531949b667f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:54:32.631050 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:32.631027 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893e7840-ce60-4a01-9120-5531949b667f-kube-api-access-r58pz" pod "893e7840-ce60-4a01-9120-5531949b667f" (UID: "893e7840-ce60-4a01-9120-5531949b667f"). InnerVolumeSpecName "kube-api-access-r58pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:54:32.734086 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:32.734035 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r58pz\" (UniqueName: \"kubernetes.io/projected/893e7840-ce60-4a01-9120-5531949b667f-kube-api-access-r58pz\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:54:32.734334 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:32.734312 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893e7840-ce60-4a01-9120-5531949b667f-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:54:33.194353 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:33.194311 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" event={"ID":"893e7840-ce60-4a01-9120-5531949b667f","Type":"ContainerDied","Data":"29e540bcde2d0c563a67cf92a7e0e5d19b0a8f9767498607c337d39f0558cf6c"} Nov 03 15:54:33.194535 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:33.194380 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29e540bcde2d0c563a67cf92a7e0e5d19b0a8f9767498607c337d39f0558cf6c" Nov 03 15:54:33.194535 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:33.194327 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62" Nov 03 15:54:38.702091 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:54:38.702055 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:54:41.050519 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:41.050485 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct"] Nov 03 15:54:41.100209 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:41.100173 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7t7ct"] Nov 03 15:54:42.678335 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:42.678302 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49004735-0202-4237-be71-7bad208a6589" path="/var/lib/kubelet/pods/49004735-0202-4237-be71-7bad208a6589/volumes" Nov 03 15:54:52.964878 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:52.964830 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-2sbg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:54:52.965687 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:54:52.965652 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" podUID="871baa2b-806e-4033-a553-277c509a3c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:55:03.219643 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.219605 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62"] Nov 03 15:55:03.268798 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.268760 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fhw62"] Nov 03 15:55:03.344375 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.344330 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv"] Nov 03 15:55:03.345010 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.344989 4290 cpu_manager.go:414] "containerMap: removing container" podUID="893e7840-ce60-4a01-9120-5531949b667f" containerName="revoke-keys" Nov 03 15:55:03.345129 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.345011 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893e7840-ce60-4a01-9120-5531949b667f" containerName="revoke-keys" Nov 03 15:55:03.345189 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.345134 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="893e7840-ce60-4a01-9120-5531949b667f" containerName="revoke-keys" Nov 03 15:55:03.348712 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.348671 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" Nov 03 15:55:03.403939 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.403890 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:55:03.403939 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.403920 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:55:03.403939 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.403922 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:55:03.404344 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.404311 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:55:03.408728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.408683 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv"] Nov 03 15:55:03.438140 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.438112 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8g46\" (UniqueName: \"kubernetes.io/projected/88b0d620-67c8-4b24-972d-f4f9d48b91ba-kube-api-access-d8g46\") pod \"maas-api-revoke-sub-3d4156f1-vc8mv\" (UID: \"88b0d620-67c8-4b24-972d-f4f9d48b91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" Nov 03 15:55:03.438322 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.438245 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88b0d620-67c8-4b24-972d-f4f9d48b91ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vc8mv\" (UID: \"88b0d620-67c8-4b24-972d-f4f9d48b91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" Nov 03 15:55:03.438322 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.438276 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88b0d620-67c8-4b24-972d-f4f9d48b91ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vc8mv\" (UID: \"88b0d620-67c8-4b24-972d-f4f9d48b91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" Nov 03 15:55:03.539564 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.539477 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88b0d620-67c8-4b24-972d-f4f9d48b91ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vc8mv\" (UID: \"88b0d620-67c8-4b24-972d-f4f9d48b91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" Nov 03 15:55:03.539564 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.539520 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88b0d620-67c8-4b24-972d-f4f9d48b91ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vc8mv\" (UID: \"88b0d620-67c8-4b24-972d-f4f9d48b91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" Nov 03 15:55:03.539816 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.539574 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8g46\" (UniqueName: \"kubernetes.io/projected/88b0d620-67c8-4b24-972d-f4f9d48b91ba-kube-api-access-d8g46\") pod \"maas-api-revoke-sub-3d4156f1-vc8mv\" (UID: \"88b0d620-67c8-4b24-972d-f4f9d48b91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" Nov 03 15:55:03.540320 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.540296 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88b0d620-67c8-4b24-972d-f4f9d48b91ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vc8mv\" (UID: \"88b0d620-67c8-4b24-972d-f4f9d48b91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" Nov 03 15:55:03.542036 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.542018 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88b0d620-67c8-4b24-972d-f4f9d48b91ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vc8mv\" (UID: \"88b0d620-67c8-4b24-972d-f4f9d48b91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" Nov 03 15:55:03.601567 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.601536 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8g46\" (UniqueName: \"kubernetes.io/projected/88b0d620-67c8-4b24-972d-f4f9d48b91ba-kube-api-access-d8g46\") pod \"maas-api-revoke-sub-3d4156f1-vc8mv\" (UID: \"88b0d620-67c8-4b24-972d-f4f9d48b91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" Nov 03 15:55:03.663841 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:03.663793 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" Nov 03 15:55:04.054801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:04.054735 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv"] Nov 03 15:55:04.055119 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:55:04.055094 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b0d620_67c8_4b24_972d_f4f9d48b91ba.slice/crio-10823f18851cf8988df908ed6c420a7805ac7d04e4196fecd3865306d99a520b WatchSource:0}: Error finding container 10823f18851cf8988df908ed6c420a7805ac7d04e4196fecd3865306d99a520b: Status 404 returned error can't find the container with id 10823f18851cf8988df908ed6c420a7805ac7d04e4196fecd3865306d99a520b Nov 03 15:55:04.493663 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:04.493623 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" event={"ID":"88b0d620-67c8-4b24-972d-f4f9d48b91ba","Type":"ContainerStarted","Data":"e2e06ff09bf8171e0e5fb9a5a3c82386a67e1100c11d87934c88abe60072414d"} Nov 03 15:55:04.494124 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:04.493729 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" event={"ID":"88b0d620-67c8-4b24-972d-f4f9d48b91ba","Type":"ContainerStarted","Data":"10823f18851cf8988df908ed6c420a7805ac7d04e4196fecd3865306d99a520b"} Nov 03 15:55:04.610490 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:04.610387 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" podStartSLOduration=1.610369127 podStartE2EDuration="1.610369127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:55:04.609485325 +0000 UTC m=+2812.040443130" watchObservedRunningTime="2026-11-03 15:55:04.610369127 +0000 UTC m=+2812.041326934" Nov 03 15:55:04.677089 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:04.677058 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="893e7840-ce60-4a01-9120-5531949b667f" path="/var/lib/kubelet/pods/893e7840-ce60-4a01-9120-5531949b667f/volumes" Nov 03 15:55:05.502876 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:05.502837 4290 generic.go:356] "Generic (PLEG): container finished" podID="88b0d620-67c8-4b24-972d-f4f9d48b91ba" containerID="e2e06ff09bf8171e0e5fb9a5a3c82386a67e1100c11d87934c88abe60072414d" exitCode=0 Nov 03 15:55:05.502876 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:05.502881 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" event={"ID":"88b0d620-67c8-4b24-972d-f4f9d48b91ba","Type":"ContainerDied","Data":"e2e06ff09bf8171e0e5fb9a5a3c82386a67e1100c11d87934c88abe60072414d"} Nov 03 15:55:06.647148 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:06.647120 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" Nov 03 15:55:06.784445 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:06.784364 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88b0d620-67c8-4b24-972d-f4f9d48b91ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88b0d620-67c8-4b24-972d-f4f9d48b91ba-maas-api-xks-ca\") pod \"88b0d620-67c8-4b24-972d-f4f9d48b91ba\" (UID: \"88b0d620-67c8-4b24-972d-f4f9d48b91ba\") " Nov 03 15:55:06.784626 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:06.784461 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b0d620-67c8-4b24-972d-f4f9d48b91ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88b0d620-67c8-4b24-972d-f4f9d48b91ba-maas-api-openshift-ca\") pod \"88b0d620-67c8-4b24-972d-f4f9d48b91ba\" (UID: \"88b0d620-67c8-4b24-972d-f4f9d48b91ba\") " Nov 03 15:55:06.784626 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:06.784555 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b0d620-67c8-4b24-972d-f4f9d48b91ba-kube-api-access-d8g46\" (UniqueName: \"kubernetes.io/projected/88b0d620-67c8-4b24-972d-f4f9d48b91ba-kube-api-access-d8g46\") pod \"88b0d620-67c8-4b24-972d-f4f9d48b91ba\" (UID: \"88b0d620-67c8-4b24-972d-f4f9d48b91ba\") " Nov 03 15:55:06.784865 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:06.784844 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b0d620-67c8-4b24-972d-f4f9d48b91ba-maas-api-openshift-ca" pod "88b0d620-67c8-4b24-972d-f4f9d48b91ba" (UID: "88b0d620-67c8-4b24-972d-f4f9d48b91ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:55:06.785741 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:06.785340 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88b0d620-67c8-4b24-972d-f4f9d48b91ba-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:55:06.787111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:06.787084 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b0d620-67c8-4b24-972d-f4f9d48b91ba-kube-api-access-d8g46" pod "88b0d620-67c8-4b24-972d-f4f9d48b91ba" (UID: "88b0d620-67c8-4b24-972d-f4f9d48b91ba"). InnerVolumeSpecName "kube-api-access-d8g46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:55:06.787111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:06.787100 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b0d620-67c8-4b24-972d-f4f9d48b91ba-maas-api-xks-ca" pod "88b0d620-67c8-4b24-972d-f4f9d48b91ba" (UID: "88b0d620-67c8-4b24-972d-f4f9d48b91ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:55:06.886851 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:06.886819 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88b0d620-67c8-4b24-972d-f4f9d48b91ba-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:55:06.886851 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:06.886851 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8g46\" (UniqueName: \"kubernetes.io/projected/88b0d620-67c8-4b24-972d-f4f9d48b91ba-kube-api-access-d8g46\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:55:07.518343 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:07.518304 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" Nov 03 15:55:07.518535 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:07.518347 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv" event={"ID":"88b0d620-67c8-4b24-972d-f4f9d48b91ba","Type":"ContainerDied","Data":"10823f18851cf8988df908ed6c420a7805ac7d04e4196fecd3865306d99a520b"} Nov 03 15:55:07.518535 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:07.518399 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10823f18851cf8988df908ed6c420a7805ac7d04e4196fecd3865306d99a520b" Nov 03 15:55:08.704628 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:55:08.704540 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:55:18.215326 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:18.215281 4290 scope.go:122] "RemoveContainer" containerID="b6ce3f82f3eb29dd8b3c027522dca97c2a3aaae2ec2d66fb242005e2938bb139" Nov 03 15:55:18.225456 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:18.225436 4290 scope.go:122] "RemoveContainer" containerID="dcec3bfe6f55469972c4a8bfef4f7ae07d8c20d35025c6c50a1a3d802b6b1fdb" Nov 03 15:55:38.704009 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:55:38.703915 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:55:50.902521 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:50.902486 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl"] Nov 03 15:55:50.903026 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:50.902994 4290 cpu_manager.go:414] "containerMap: removing container" podUID="88b0d620-67c8-4b24-972d-f4f9d48b91ba" containerName="revoke-keys" Nov 03 15:55:50.903026 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:50.903007 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b0d620-67c8-4b24-972d-f4f9d48b91ba" containerName="revoke-keys" Nov 03 15:55:50.903120 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:50.903100 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b0d620-67c8-4b24-972d-f4f9d48b91ba" containerName="revoke-keys" Nov 03 15:55:50.906044 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:50.906025 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" Nov 03 15:55:50.994148 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:50.994115 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-l5txl\" (UID: \"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" Nov 03 15:55:50.994312 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:50.994241 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5wxm\" (UniqueName: \"kubernetes.io/projected/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-kube-api-access-v5wxm\") pod \"maas-api-revoke-sub-e2fc9c5b-l5txl\" (UID: \"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" Nov 03 15:55:50.994312 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:50.994295 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-l5txl\" (UID: \"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" Nov 03 15:55:51.022423 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.022387 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:55:51.022580 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.022475 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:55:51.022580 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.022487 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:55:51.022684 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.022625 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:55:51.026412 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.026390 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl"] Nov 03 15:55:51.079406 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.079364 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf"] Nov 03 15:55:51.082396 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.082367 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-k9tnf"] Nov 03 15:55:51.095183 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.095151 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-l5txl\" (UID: \"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" Nov 03 15:55:51.095307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.095234 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5wxm\" (UniqueName: \"kubernetes.io/projected/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-kube-api-access-v5wxm\") pod \"maas-api-revoke-sub-e2fc9c5b-l5txl\" (UID: \"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" Nov 03 15:55:51.095307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.095266 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-l5txl\" (UID: \"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" Nov 03 15:55:51.095796 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.095777 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-l5txl\" (UID: \"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" Nov 03 15:55:51.097672 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.097651 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-l5txl\" (UID: \"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" Nov 03 15:55:51.157601 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.157538 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5wxm\" (UniqueName: \"kubernetes.io/projected/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-kube-api-access-v5wxm\") pod \"maas-api-revoke-sub-e2fc9c5b-l5txl\" (UID: \"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" Nov 03 15:55:51.219574 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.219543 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" Nov 03 15:55:51.600801 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.600772 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl"] Nov 03 15:55:51.601549 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:55:51.601510 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec3f9fe_f02b_4b99_83fb_42f2353bfc88.slice/crio-8ce89cf8cadde9fb5f968a5e6b47a83de4230715428e692570c09ea791db2fce WatchSource:0}: Error finding container 8ce89cf8cadde9fb5f968a5e6b47a83de4230715428e692570c09ea791db2fce: Status 404 returned error can't find the container with id 8ce89cf8cadde9fb5f968a5e6b47a83de4230715428e692570c09ea791db2fce Nov 03 15:55:51.871233 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.871204 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" event={"ID":"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88","Type":"ContainerStarted","Data":"31e87b5ac9dba59438044bbfd5aba786048c9eaa0c62e8bbf20f0d439a70b66a"} Nov 03 15:55:51.871369 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.871257 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" event={"ID":"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88","Type":"ContainerStarted","Data":"8ce89cf8cadde9fb5f968a5e6b47a83de4230715428e692570c09ea791db2fce"} Nov 03 15:55:51.989085 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:51.989030 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" podStartSLOduration=1.9890155200000001 podStartE2EDuration="1.98901552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:55:51.987548649 +0000 UTC m=+2859.418506452" watchObservedRunningTime="2026-11-03 15:55:51.98901552 +0000 UTC m=+2859.419973326" Nov 03 15:55:52.677765 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:52.677727 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf0012a-424e-455e-86a5-66776eb99c93" path="/var/lib/kubelet/pods/3cf0012a-424e-455e-86a5-66776eb99c93/volumes" Nov 03 15:55:52.879242 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:52.879208 4290 generic.go:356] "Generic (PLEG): container finished" podID="9ec3f9fe-f02b-4b99-83fb-42f2353bfc88" containerID="31e87b5ac9dba59438044bbfd5aba786048c9eaa0c62e8bbf20f0d439a70b66a" exitCode=0 Nov 03 15:55:52.879443 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:52.879290 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" event={"ID":"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88","Type":"ContainerDied","Data":"31e87b5ac9dba59438044bbfd5aba786048c9eaa0c62e8bbf20f0d439a70b66a"} Nov 03 15:55:54.023046 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:54.023024 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" Nov 03 15:55:54.127383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:54.127348 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-maas-api-openshift-ca\") pod \"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88\" (UID: \"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88\") " Nov 03 15:55:54.127562 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:54.127460 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-kube-api-access-v5wxm\" (UniqueName: \"kubernetes.io/projected/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-kube-api-access-v5wxm\") pod \"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88\" (UID: \"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88\") " Nov 03 15:55:54.127562 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:54.127506 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-maas-api-xks-ca\") pod \"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88\" (UID: \"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88\") " Nov 03 15:55:54.127786 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:54.127759 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-maas-api-openshift-ca" pod "9ec3f9fe-f02b-4b99-83fb-42f2353bfc88" (UID: "9ec3f9fe-f02b-4b99-83fb-42f2353bfc88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:55:54.129798 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:54.129778 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-kube-api-access-v5wxm" pod "9ec3f9fe-f02b-4b99-83fb-42f2353bfc88" (UID: "9ec3f9fe-f02b-4b99-83fb-42f2353bfc88"). InnerVolumeSpecName "kube-api-access-v5wxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:55:54.129798 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:54.129756 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-maas-api-xks-ca" pod "9ec3f9fe-f02b-4b99-83fb-42f2353bfc88" (UID: "9ec3f9fe-f02b-4b99-83fb-42f2353bfc88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:55:54.229205 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:54.229118 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5wxm\" (UniqueName: \"kubernetes.io/projected/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-kube-api-access-v5wxm\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:55:54.229205 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:54.229149 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:55:54.229205 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:54.229165 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:55:54.894363 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:54.894333 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" Nov 03 15:55:54.894558 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:54.894380 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl" event={"ID":"9ec3f9fe-f02b-4b99-83fb-42f2353bfc88","Type":"ContainerDied","Data":"8ce89cf8cadde9fb5f968a5e6b47a83de4230715428e692570c09ea791db2fce"} Nov 03 15:55:54.894558 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:55:54.894434 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ce89cf8cadde9fb5f968a5e6b47a83de4230715428e692570c09ea791db2fce" Nov 03 15:56:01.652243 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.652211 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps"] Nov 03 15:56:01.652757 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.652746 4290 cpu_manager.go:414] "containerMap: removing container" podUID="9ec3f9fe-f02b-4b99-83fb-42f2353bfc88" containerName="revoke-keys" Nov 03 15:56:01.652811 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.652758 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec3f9fe-f02b-4b99-83fb-42f2353bfc88" containerName="revoke-keys" Nov 03 15:56:01.652858 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.652848 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec3f9fe-f02b-4b99-83fb-42f2353bfc88" containerName="revoke-keys" Nov 03 15:56:01.656287 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.656267 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" Nov 03 15:56:01.703604 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.703573 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hv78\" (UniqueName: \"kubernetes.io/projected/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-kube-api-access-5hv78\") pod \"maas-api-revoke-sub-5a29efb3-cfqps\" (UID: \"dd6a20c5-0cde-44ee-9047-a85e2a036a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" Nov 03 15:56:01.703826 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.703642 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-cfqps\" (UID: \"dd6a20c5-0cde-44ee-9047-a85e2a036a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" Nov 03 15:56:01.703826 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.703756 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-cfqps\" (UID: \"dd6a20c5-0cde-44ee-9047-a85e2a036a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" Nov 03 15:56:01.709403 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.709379 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:56:01.759379 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.759347 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:56:01.759379 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.759347 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:56:01.759578 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.759360 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:56:01.763198 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.763176 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps"] Nov 03 15:56:01.805046 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.805018 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hv78\" (UniqueName: \"kubernetes.io/projected/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-kube-api-access-5hv78\") pod \"maas-api-revoke-sub-5a29efb3-cfqps\" (UID: \"dd6a20c5-0cde-44ee-9047-a85e2a036a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" Nov 03 15:56:01.805180 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.805076 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-cfqps\" (UID: \"dd6a20c5-0cde-44ee-9047-a85e2a036a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" Nov 03 15:56:01.805180 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.805116 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-cfqps\" (UID: \"dd6a20c5-0cde-44ee-9047-a85e2a036a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" Nov 03 15:56:01.805759 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.805737 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-cfqps\" (UID: \"dd6a20c5-0cde-44ee-9047-a85e2a036a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" Nov 03 15:56:01.807841 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.807822 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-cfqps\" (UID: \"dd6a20c5-0cde-44ee-9047-a85e2a036a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" Nov 03 15:56:01.867487 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.867460 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hv78\" (UniqueName: \"kubernetes.io/projected/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-kube-api-access-5hv78\") pod \"maas-api-revoke-sub-5a29efb3-cfqps\" (UID: \"dd6a20c5-0cde-44ee-9047-a85e2a036a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" Nov 03 15:56:01.970334 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:01.970247 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" Nov 03 15:56:02.351032 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:02.350982 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps"] Nov 03 15:56:02.351537 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:56:02.351515 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6a20c5_0cde_44ee_9047_a85e2a036a6a.slice/crio-2ca1fb27a9cf4e9ad22b1db1c8edcaaf161129c5086242ac07fdf3fc221e2a36 WatchSource:0}: Error finding container 2ca1fb27a9cf4e9ad22b1db1c8edcaaf161129c5086242ac07fdf3fc221e2a36: Status 404 returned error can't find the container with id 2ca1fb27a9cf4e9ad22b1db1c8edcaaf161129c5086242ac07fdf3fc221e2a36 Nov 03 15:56:02.948681 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:02.948590 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" event={"ID":"dd6a20c5-0cde-44ee-9047-a85e2a036a6a","Type":"ContainerStarted","Data":"e23ec56fe4c1ce1714be0a50eb3f0e4c413b52ad7f2ce4d51cf609fa20921317"} Nov 03 15:56:02.948681 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:02.948648 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" event={"ID":"dd6a20c5-0cde-44ee-9047-a85e2a036a6a","Type":"ContainerStarted","Data":"2ca1fb27a9cf4e9ad22b1db1c8edcaaf161129c5086242ac07fdf3fc221e2a36"} Nov 03 15:56:03.066161 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:03.066105 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" podStartSLOduration=2.066090424 podStartE2EDuration="2.066090424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:56:03.065590772 +0000 UTC m=+2870.496548578" watchObservedRunningTime="2026-11-03 15:56:03.066090424 +0000 UTC m=+2870.497048229" Nov 03 15:56:03.957126 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:03.957094 4290 generic.go:356] "Generic (PLEG): container finished" podID="dd6a20c5-0cde-44ee-9047-a85e2a036a6a" containerID="e23ec56fe4c1ce1714be0a50eb3f0e4c413b52ad7f2ce4d51cf609fa20921317" exitCode=0 Nov 03 15:56:03.957641 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:03.957180 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" event={"ID":"dd6a20c5-0cde-44ee-9047-a85e2a036a6a","Type":"ContainerDied","Data":"e23ec56fe4c1ce1714be0a50eb3f0e4c413b52ad7f2ce4d51cf609fa20921317"} Nov 03 15:56:05.124634 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:05.124607 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" Nov 03 15:56:05.240788 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:05.240036 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-kube-api-access-5hv78\" (UniqueName: \"kubernetes.io/projected/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-kube-api-access-5hv78\") pod \"dd6a20c5-0cde-44ee-9047-a85e2a036a6a\" (UID: \"dd6a20c5-0cde-44ee-9047-a85e2a036a6a\") " Nov 03 15:56:05.240788 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:05.240120 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-maas-api-openshift-ca\") pod \"dd6a20c5-0cde-44ee-9047-a85e2a036a6a\" (UID: \"dd6a20c5-0cde-44ee-9047-a85e2a036a6a\") " Nov 03 15:56:05.240788 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:05.240211 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-maas-api-xks-ca\") pod \"dd6a20c5-0cde-44ee-9047-a85e2a036a6a\" (UID: \"dd6a20c5-0cde-44ee-9047-a85e2a036a6a\") " Nov 03 15:56:05.267156 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:05.259690 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-maas-api-openshift-ca" pod "dd6a20c5-0cde-44ee-9047-a85e2a036a6a" (UID: "dd6a20c5-0cde-44ee-9047-a85e2a036a6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:56:05.267156 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:05.262162 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-maas-api-xks-ca" pod "dd6a20c5-0cde-44ee-9047-a85e2a036a6a" (UID: "dd6a20c5-0cde-44ee-9047-a85e2a036a6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:56:05.273220 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:05.273172 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-kube-api-access-5hv78" pod "dd6a20c5-0cde-44ee-9047-a85e2a036a6a" (UID: "dd6a20c5-0cde-44ee-9047-a85e2a036a6a"). InnerVolumeSpecName "kube-api-access-5hv78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:56:05.342196 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:05.342155 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hv78\" (UniqueName: \"kubernetes.io/projected/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-kube-api-access-5hv78\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:56:05.342448 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:05.342428 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:56:05.342569 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:05.342554 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6a20c5-0cde-44ee-9047-a85e2a036a6a-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:56:06.001951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:06.001845 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" event={"ID":"dd6a20c5-0cde-44ee-9047-a85e2a036a6a","Type":"ContainerDied","Data":"2ca1fb27a9cf4e9ad22b1db1c8edcaaf161129c5086242ac07fdf3fc221e2a36"} Nov 03 15:56:06.001951 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:06.001916 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ca1fb27a9cf4e9ad22b1db1c8edcaaf161129c5086242ac07fdf3fc221e2a36" Nov 03 15:56:06.002270 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:06.002249 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps" Nov 03 15:56:08.703529 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:56:08.703442 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:56:18.256360 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:18.256320 4290 scope.go:122] "RemoveContainer" containerID="059fce8f8cca25743b2a4db18bed221a19ccbe9897a331889e46bd08cb8d6215" Nov 03 15:56:18.877939 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:18.877898 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv"] Nov 03 15:56:18.909644 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:18.909613 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vc8mv"] Nov 03 15:56:18.998841 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:18.998810 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb"] Nov 03 15:56:18.999313 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:18.999301 4290 cpu_manager.go:414] "containerMap: removing container" podUID="dd6a20c5-0cde-44ee-9047-a85e2a036a6a" containerName="revoke-keys" Nov 03 15:56:18.999405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:18.999315 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6a20c5-0cde-44ee-9047-a85e2a036a6a" containerName="revoke-keys" Nov 03 15:56:18.999405 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:18.999388 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6a20c5-0cde-44ee-9047-a85e2a036a6a" containerName="revoke-keys" Nov 03 15:56:19.004005 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.003986 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" Nov 03 15:56:19.059979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.059675 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:56:19.060221 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.060198 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:56:19.060221 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.060210 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:56:19.060366 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.060240 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:56:19.065666 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.065645 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb"] Nov 03 15:56:19.104866 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.104833 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/393ebecd-95aa-44e3-921d-7825c86b9bec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ts7gb\" (UID: \"393ebecd-95aa-44e3-921d-7825c86b9bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" Nov 03 15:56:19.105085 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.104978 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/393ebecd-95aa-44e3-921d-7825c86b9bec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ts7gb\" (UID: \"393ebecd-95aa-44e3-921d-7825c86b9bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" Nov 03 15:56:19.105085 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.105031 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrwhp\" (UniqueName: \"kubernetes.io/projected/393ebecd-95aa-44e3-921d-7825c86b9bec-kube-api-access-xrwhp\") pod \"maas-api-revoke-sub-3d4156f1-ts7gb\" (UID: \"393ebecd-95aa-44e3-921d-7825c86b9bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" Nov 03 15:56:19.206278 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.206193 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/393ebecd-95aa-44e3-921d-7825c86b9bec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ts7gb\" (UID: \"393ebecd-95aa-44e3-921d-7825c86b9bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" Nov 03 15:56:19.206278 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.206241 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrwhp\" (UniqueName: \"kubernetes.io/projected/393ebecd-95aa-44e3-921d-7825c86b9bec-kube-api-access-xrwhp\") pod \"maas-api-revoke-sub-3d4156f1-ts7gb\" (UID: \"393ebecd-95aa-44e3-921d-7825c86b9bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" Nov 03 15:56:19.206467 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.206286 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/393ebecd-95aa-44e3-921d-7825c86b9bec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ts7gb\" (UID: \"393ebecd-95aa-44e3-921d-7825c86b9bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" Nov 03 15:56:19.206904 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.206872 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/393ebecd-95aa-44e3-921d-7825c86b9bec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ts7gb\" (UID: \"393ebecd-95aa-44e3-921d-7825c86b9bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" Nov 03 15:56:19.208746 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.208723 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/393ebecd-95aa-44e3-921d-7825c86b9bec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ts7gb\" (UID: \"393ebecd-95aa-44e3-921d-7825c86b9bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" Nov 03 15:56:19.268599 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.268570 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrwhp\" (UniqueName: \"kubernetes.io/projected/393ebecd-95aa-44e3-921d-7825c86b9bec-kube-api-access-xrwhp\") pod \"maas-api-revoke-sub-3d4156f1-ts7gb\" (UID: \"393ebecd-95aa-44e3-921d-7825c86b9bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" Nov 03 15:56:19.316675 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.316648 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" Nov 03 15:56:19.701103 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:19.701070 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb"] Nov 03 15:56:19.701793 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:56:19.701761 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393ebecd_95aa_44e3_921d_7825c86b9bec.slice/crio-0ebe018b6d3271c949df2b77048343b7f6e3b5aabdb9800069c3ee02e80bc305 WatchSource:0}: Error finding container 0ebe018b6d3271c949df2b77048343b7f6e3b5aabdb9800069c3ee02e80bc305: Status 404 returned error can't find the container with id 0ebe018b6d3271c949df2b77048343b7f6e3b5aabdb9800069c3ee02e80bc305 Nov 03 15:56:20.144252 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:20.144212 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" event={"ID":"393ebecd-95aa-44e3-921d-7825c86b9bec","Type":"ContainerStarted","Data":"9ba664ed46c2e3eb7e4ea9a432217175b261837630d1e4f4caac8d3b88bc9db8"} Nov 03 15:56:20.144458 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:20.144300 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" event={"ID":"393ebecd-95aa-44e3-921d-7825c86b9bec","Type":"ContainerStarted","Data":"0ebe018b6d3271c949df2b77048343b7f6e3b5aabdb9800069c3ee02e80bc305"} Nov 03 15:56:20.262961 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:20.262856 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" podStartSLOduration=2.262840645 podStartE2EDuration="2.262840645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:56:20.259880486 +0000 UTC m=+2887.690838293" watchObservedRunningTime="2026-11-03 15:56:20.262840645 +0000 UTC m=+2887.693798450" Nov 03 15:56:20.679270 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:20.679235 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b0d620-67c8-4b24-972d-f4f9d48b91ba" path="/var/lib/kubelet/pods/88b0d620-67c8-4b24-972d-f4f9d48b91ba/volumes" Nov 03 15:56:21.153848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:21.153816 4290 generic.go:356] "Generic (PLEG): container finished" podID="393ebecd-95aa-44e3-921d-7825c86b9bec" containerID="9ba664ed46c2e3eb7e4ea9a432217175b261837630d1e4f4caac8d3b88bc9db8" exitCode=0 Nov 03 15:56:21.154044 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:21.153908 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" event={"ID":"393ebecd-95aa-44e3-921d-7825c86b9bec","Type":"ContainerDied","Data":"9ba664ed46c2e3eb7e4ea9a432217175b261837630d1e4f4caac8d3b88bc9db8"} Nov 03 15:56:22.051903 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:22.051870 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6"] Nov 03 15:56:22.102051 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:22.102020 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpjm6"] Nov 03 15:56:22.317543 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:22.317472 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" Nov 03 15:56:22.439193 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:22.439154 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393ebecd-95aa-44e3-921d-7825c86b9bec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/393ebecd-95aa-44e3-921d-7825c86b9bec-maas-api-openshift-ca\") pod \"393ebecd-95aa-44e3-921d-7825c86b9bec\" (UID: \"393ebecd-95aa-44e3-921d-7825c86b9bec\") " Nov 03 15:56:22.439388 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:22.439256 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393ebecd-95aa-44e3-921d-7825c86b9bec-kube-api-access-xrwhp\" (UniqueName: \"kubernetes.io/projected/393ebecd-95aa-44e3-921d-7825c86b9bec-kube-api-access-xrwhp\") pod \"393ebecd-95aa-44e3-921d-7825c86b9bec\" (UID: \"393ebecd-95aa-44e3-921d-7825c86b9bec\") " Nov 03 15:56:22.439388 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:22.439324 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393ebecd-95aa-44e3-921d-7825c86b9bec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/393ebecd-95aa-44e3-921d-7825c86b9bec-maas-api-xks-ca\") pod \"393ebecd-95aa-44e3-921d-7825c86b9bec\" (UID: \"393ebecd-95aa-44e3-921d-7825c86b9bec\") " Nov 03 15:56:22.439579 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:22.439551 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393ebecd-95aa-44e3-921d-7825c86b9bec-maas-api-openshift-ca" pod "393ebecd-95aa-44e3-921d-7825c86b9bec" (UID: "393ebecd-95aa-44e3-921d-7825c86b9bec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:56:22.439769 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:22.439749 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/393ebecd-95aa-44e3-921d-7825c86b9bec-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:56:22.441790 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:22.441756 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393ebecd-95aa-44e3-921d-7825c86b9bec-kube-api-access-xrwhp" pod "393ebecd-95aa-44e3-921d-7825c86b9bec" (UID: "393ebecd-95aa-44e3-921d-7825c86b9bec"). InnerVolumeSpecName "kube-api-access-xrwhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:56:22.443876 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:22.443800 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393ebecd-95aa-44e3-921d-7825c86b9bec-maas-api-xks-ca" pod "393ebecd-95aa-44e3-921d-7825c86b9bec" (UID: "393ebecd-95aa-44e3-921d-7825c86b9bec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:56:22.540577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:22.540539 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrwhp\" (UniqueName: \"kubernetes.io/projected/393ebecd-95aa-44e3-921d-7825c86b9bec-kube-api-access-xrwhp\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:56:22.540577 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:22.540575 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/393ebecd-95aa-44e3-921d-7825c86b9bec-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:56:22.677597 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:22.677564 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a" path="/var/lib/kubelet/pods/8f31c536-5bd0-4e6e-a9e7-b0be825e3a8a/volumes" Nov 03 15:56:23.168991 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:23.168951 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" event={"ID":"393ebecd-95aa-44e3-921d-7825c86b9bec","Type":"ContainerDied","Data":"0ebe018b6d3271c949df2b77048343b7f6e3b5aabdb9800069c3ee02e80bc305"} Nov 03 15:56:23.169423 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:23.169001 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ebe018b6d3271c949df2b77048343b7f6e3b5aabdb9800069c3ee02e80bc305" Nov 03 15:56:23.169423 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:23.168959 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb" Nov 03 15:56:35.074121 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:35.074078 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx"] Nov 03 15:56:35.114846 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:35.114811 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vkrxx"] Nov 03 15:56:36.677944 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:56:36.677908 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baa562a9-e6cd-4d77-9733-a1dd74490a3e" path="/var/lib/kubelet/pods/baa562a9-e6cd-4d77-9733-a1dd74490a3e/volumes" Nov 03 15:56:38.701383 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:56:38.701334 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:57:08.703308 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:57:08.703220 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:57:11.056283 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:11.056248 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv"] Nov 03 15:57:11.103795 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:11.103764 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vmfvv"] Nov 03 15:57:12.677578 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:12.677547 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6" path="/var/lib/kubelet/pods/98c0d14a-e9a1-4fc9-a3d7-d0611a350bc6/volumes" Nov 03 15:57:18.285218 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:18.285185 4290 scope.go:122] "RemoveContainer" containerID="ce8488e186103ab9fe25adb68d328eda373b7afa3bc67e38dfa8b67eccb703d3" Nov 03 15:57:21.053960 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:21.053924 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c"] Nov 03 15:57:21.103235 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:21.103197 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nzz4c"] Nov 03 15:57:22.677090 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:22.677057 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4521bde-6c96-45a3-9f02-14129edc3c5d" path="/var/lib/kubelet/pods/d4521bde-6c96-45a3-9f02-14129edc3c5d/volumes" Nov 03 15:57:25.186751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.186716 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6"] Nov 03 15:57:25.187223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.187210 4290 cpu_manager.go:414] "containerMap: removing container" podUID="393ebecd-95aa-44e3-921d-7825c86b9bec" containerName="revoke-keys" Nov 03 15:57:25.187268 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.187224 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393ebecd-95aa-44e3-921d-7825c86b9bec" containerName="revoke-keys" Nov 03 15:57:25.187349 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.187307 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="393ebecd-95aa-44e3-921d-7825c86b9bec" containerName="revoke-keys" Nov 03 15:57:25.191655 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.191635 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.262901 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.262866 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6"] Nov 03 15:57:25.296517 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.296482 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d9d854\"/\"openshift-service-ca.crt\"" Nov 03 15:57:25.296517 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.296480 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d9d854\"/\"simulator-single-main-d9d854-kserve-self-signed-certs\"" Nov 03 15:57:25.296767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.296564 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d9d854\"/\"kube-root-ca.crt\"" Nov 03 15:57:25.296767 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.296728 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d9d854\"/\"default-dockercfg-mvxz6\"" Nov 03 15:57:25.341320 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.341277 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-tmp-dir\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.341511 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.341419 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-model-cache\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.341511 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.341457 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-dshm\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.341604 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.341576 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-home\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.341639 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.341620 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e31080a-88a6-4908-a759-910fb44ac101-tls-certs\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.341728 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.341715 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp9gd\" (UniqueName: \"kubernetes.io/projected/3e31080a-88a6-4908-a759-910fb44ac101-kube-api-access-fp9gd\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.442828 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.442738 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-tmp-dir\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.442828 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.442810 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-model-cache\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.443037 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.442834 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-dshm\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.443037 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.442874 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-home\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.443037 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.442905 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e31080a-88a6-4908-a759-910fb44ac101-tls-certs\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.443037 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.442959 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp9gd\" (UniqueName: \"kubernetes.io/projected/3e31080a-88a6-4908-a759-910fb44ac101-kube-api-access-fp9gd\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.443298 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.443274 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-model-cache\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.443355 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.443334 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-home\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.443430 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.443392 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-tmp-dir\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.445378 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.445359 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-dshm\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.445636 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.445618 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e31080a-88a6-4908-a759-910fb44ac101-tls-certs\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.508300 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.508259 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp9gd\" (UniqueName: \"kubernetes.io/projected/3e31080a-88a6-4908-a759-910fb44ac101-kube-api-access-fp9gd\") pod \"simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:25.803886 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:25.803795 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:26.187908 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:26.187876 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6"] Nov 03 15:57:26.188990 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:57:26.188962 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e31080a_88a6_4908_a759_910fb44ac101.slice/crio-6d3d0784a90bc294a31be092095374f275e2f0176b59581c22b82828d627e519 WatchSource:0}: Error finding container 6d3d0784a90bc294a31be092095374f275e2f0176b59581c22b82828d627e519: Status 404 returned error can't find the container with id 6d3d0784a90bc294a31be092095374f275e2f0176b59581c22b82828d627e519 Nov 03 15:57:26.668899 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:26.668860 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" event={"ID":"3e31080a-88a6-4908-a759-910fb44ac101","Type":"ContainerStarted","Data":"6c688352a1901e7205008932e4e03521d1acc863b5040dcb1b44e34dc04b1fb7"} Nov 03 15:57:26.668899 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:26.668910 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" event={"ID":"3e31080a-88a6-4908-a759-910fb44ac101","Type":"ContainerStarted","Data":"6d3d0784a90bc294a31be092095374f275e2f0176b59581c22b82828d627e519"} Nov 03 15:57:26.790439 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:26.790385 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" podStartSLOduration=1.790369756 podStartE2EDuration="1.790369756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:57:26.788850459 +0000 UTC m=+2954.219808266" watchObservedRunningTime="2026-11-03 15:57:26.790369756 +0000 UTC m=+2954.221327592" Nov 03 15:57:26.804582 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:26.804550 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:35.804148 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:35.804114 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:35.817002 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:35.816974 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:36.753228 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:36.753193 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:57:38.702625 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:57:38.702585 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:57:59.052047 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:59.051998 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7"] Nov 03 15:57:59.101843 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:57:59.101798 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-47xl7"] Nov 03 15:58:00.677478 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:00.677444 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd533f31-7e0f-49a0-838a-312c84eeebb5" path="/var/lib/kubelet/pods/cd533f31-7e0f-49a0-838a-312c84eeebb5/volumes" Nov 03 15:58:08.705373 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:58:08.705281 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:58:09.492988 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.492957 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs"] Nov 03 15:58:09.497269 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.497249 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" Nov 03 15:58:09.555980 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.555951 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs"] Nov 03 15:58:09.601636 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.601590 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:58:09.601848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.601665 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:58:09.601848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.601725 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:58:09.601848 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.601730 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:58:09.623662 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.623637 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c66347-024c-4bf6-a7cf-dafe50931382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ca95f6-7f2cs\" (UID: \"65c66347-024c-4bf6-a7cf-dafe50931382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" Nov 03 15:58:09.623770 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.623677 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ctc8\" (UniqueName: \"kubernetes.io/projected/65c66347-024c-4bf6-a7cf-dafe50931382-kube-api-access-6ctc8\") pod \"maas-api-revoke-sub-c0ca95f6-7f2cs\" (UID: \"65c66347-024c-4bf6-a7cf-dafe50931382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" Nov 03 15:58:09.623825 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.623808 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c66347-024c-4bf6-a7cf-dafe50931382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ca95f6-7f2cs\" (UID: \"65c66347-024c-4bf6-a7cf-dafe50931382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" Nov 03 15:58:09.725383 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.725345 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c66347-024c-4bf6-a7cf-dafe50931382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ca95f6-7f2cs\" (UID: \"65c66347-024c-4bf6-a7cf-dafe50931382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" Nov 03 15:58:09.725838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.725392 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ctc8\" (UniqueName: \"kubernetes.io/projected/65c66347-024c-4bf6-a7cf-dafe50931382-kube-api-access-6ctc8\") pod \"maas-api-revoke-sub-c0ca95f6-7f2cs\" (UID: \"65c66347-024c-4bf6-a7cf-dafe50931382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" Nov 03 15:58:09.725838 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.725435 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c66347-024c-4bf6-a7cf-dafe50931382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ca95f6-7f2cs\" (UID: \"65c66347-024c-4bf6-a7cf-dafe50931382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" Nov 03 15:58:09.726178 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.726153 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c66347-024c-4bf6-a7cf-dafe50931382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ca95f6-7f2cs\" (UID: \"65c66347-024c-4bf6-a7cf-dafe50931382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" Nov 03 15:58:09.727956 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.727937 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c66347-024c-4bf6-a7cf-dafe50931382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ca95f6-7f2cs\" (UID: \"65c66347-024c-4bf6-a7cf-dafe50931382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" Nov 03 15:58:09.787627 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.787557 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ctc8\" (UniqueName: \"kubernetes.io/projected/65c66347-024c-4bf6-a7cf-dafe50931382-kube-api-access-6ctc8\") pod \"maas-api-revoke-sub-c0ca95f6-7f2cs\" (UID: \"65c66347-024c-4bf6-a7cf-dafe50931382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" Nov 03 15:58:09.811531 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:09.811501 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" Nov 03 15:58:10.191998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:10.191968 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs"] Nov 03 15:58:10.192834 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:58:10.192808 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c66347_024c_4bf6_a7cf_dafe50931382.slice/crio-e66a4e108371e6916ed2d6af67ff860877964b249514de481965f03b7f504ec1 WatchSource:0}: Error finding container e66a4e108371e6916ed2d6af67ff860877964b249514de481965f03b7f504ec1: Status 404 returned error can't find the container with id e66a4e108371e6916ed2d6af67ff860877964b249514de481965f03b7f504ec1 Nov 03 15:58:10.995187 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:10.995143 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" event={"ID":"65c66347-024c-4bf6-a7cf-dafe50931382","Type":"ContainerStarted","Data":"5ff12f9eaf1a38d2540d8463f6e584c6444e95e2ceb9e0ddbc8e1de849dd27c6"} Nov 03 15:58:10.995680 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:10.995208 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" event={"ID":"65c66347-024c-4bf6-a7cf-dafe50931382","Type":"ContainerStarted","Data":"e66a4e108371e6916ed2d6af67ff860877964b249514de481965f03b7f504ec1"} Nov 03 15:58:11.112307 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:11.112243 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" podStartSLOduration=2.112222595 podStartE2EDuration="2.112222595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:58:11.11174969 +0000 UTC m=+2998.542707506" watchObservedRunningTime="2026-11-03 15:58:11.112222595 +0000 UTC m=+2998.543180415" Nov 03 15:58:12.004132 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:12.004094 4290 generic.go:356] "Generic (PLEG): container finished" podID="65c66347-024c-4bf6-a7cf-dafe50931382" containerID="5ff12f9eaf1a38d2540d8463f6e584c6444e95e2ceb9e0ddbc8e1de849dd27c6" exitCode=0 Nov 03 15:58:12.004581 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:12.004180 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" event={"ID":"65c66347-024c-4bf6-a7cf-dafe50931382","Type":"ContainerDied","Data":"5ff12f9eaf1a38d2540d8463f6e584c6444e95e2ceb9e0ddbc8e1de849dd27c6"} Nov 03 15:58:13.148790 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:13.148758 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" Nov 03 15:58:13.265972 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:13.264997 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65c66347-024c-4bf6-a7cf-dafe50931382-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c66347-024c-4bf6-a7cf-dafe50931382-maas-api-xks-ca\") pod \"65c66347-024c-4bf6-a7cf-dafe50931382\" (UID: \"65c66347-024c-4bf6-a7cf-dafe50931382\") " Nov 03 15:58:13.266147 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:13.266098 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c66347-024c-4bf6-a7cf-dafe50931382-kube-api-access-6ctc8\" (UniqueName: \"kubernetes.io/projected/65c66347-024c-4bf6-a7cf-dafe50931382-kube-api-access-6ctc8\") pod \"65c66347-024c-4bf6-a7cf-dafe50931382\" (UID: \"65c66347-024c-4bf6-a7cf-dafe50931382\") " Nov 03 15:58:13.266214 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:13.266146 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c66347-024c-4bf6-a7cf-dafe50931382-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c66347-024c-4bf6-a7cf-dafe50931382-maas-api-openshift-ca\") pod \"65c66347-024c-4bf6-a7cf-dafe50931382\" (UID: \"65c66347-024c-4bf6-a7cf-dafe50931382\") " Nov 03 15:58:13.266887 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:13.266849 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c66347-024c-4bf6-a7cf-dafe50931382-maas-api-openshift-ca" pod "65c66347-024c-4bf6-a7cf-dafe50931382" (UID: "65c66347-024c-4bf6-a7cf-dafe50931382"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:58:13.268114 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:13.268084 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65c66347-024c-4bf6-a7cf-dafe50931382-maas-api-xks-ca" pod "65c66347-024c-4bf6-a7cf-dafe50931382" (UID: "65c66347-024c-4bf6-a7cf-dafe50931382"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:58:13.270784 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:13.270759 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c66347-024c-4bf6-a7cf-dafe50931382-kube-api-access-6ctc8" pod "65c66347-024c-4bf6-a7cf-dafe50931382" (UID: "65c66347-024c-4bf6-a7cf-dafe50931382"). InnerVolumeSpecName "kube-api-access-6ctc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:58:13.367485 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:13.367452 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c66347-024c-4bf6-a7cf-dafe50931382-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:58:13.367485 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:13.367486 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ctc8\" (UniqueName: \"kubernetes.io/projected/65c66347-024c-4bf6-a7cf-dafe50931382-kube-api-access-6ctc8\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:58:13.367777 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:13.367501 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c66347-024c-4bf6-a7cf-dafe50931382-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:58:13.370515 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:13.370487 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:58:13.373184 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:13.373158 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:58:13.381185 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:13.381164 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 15:58:13.384425 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:13.384400 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 15:58:14.020192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:14.020159 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" Nov 03 15:58:14.020404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:14.020198 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs" event={"ID":"65c66347-024c-4bf6-a7cf-dafe50931382","Type":"ContainerDied","Data":"e66a4e108371e6916ed2d6af67ff860877964b249514de481965f03b7f504ec1"} Nov 03 15:58:14.020404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:14.020247 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e66a4e108371e6916ed2d6af67ff860877964b249514de481965f03b7f504ec1" Nov 03 15:58:18.315609 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:18.315576 4290 scope.go:122] "RemoveContainer" containerID="ce4efb0924133640605172068b479b936d3a012010942794ebe6f0df95f3d667" Nov 03 15:58:18.326266 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:18.326241 4290 scope.go:122] "RemoveContainer" containerID="c694fc34e0a0b5c40fb7a4f847f8022360eb3cc817647a798e54424029c35b46" Nov 03 15:58:18.335668 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:18.335646 4290 scope.go:122] "RemoveContainer" containerID="4f49d601fe89940cea93ee884dbf184e734789c3f207e145bca4ac5e337b51f1" Nov 03 15:58:18.345623 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:18.345600 4290 scope.go:122] "RemoveContainer" containerID="b05940e53e5689555c1013f6331ac066203a52a48ba8eabe4e0dfdf6e48100a4" Nov 03 15:58:21.837709 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.837654 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc"] Nov 03 15:58:21.838183 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.838168 4290 cpu_manager.go:414] "containerMap: removing container" podUID="65c66347-024c-4bf6-a7cf-dafe50931382" containerName="revoke-keys" Nov 03 15:58:21.838233 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.838184 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c66347-024c-4bf6-a7cf-dafe50931382" containerName="revoke-keys" Nov 03 15:58:21.838295 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.838286 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c66347-024c-4bf6-a7cf-dafe50931382" containerName="revoke-keys" Nov 03 15:58:21.843256 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.843231 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" Nov 03 15:58:21.850543 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.850431 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc\" (UID: \"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" Nov 03 15:58:21.850543 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.850481 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc\" (UID: \"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" Nov 03 15:58:21.850684 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.850555 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grmv5\" (UniqueName: \"kubernetes.io/projected/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-kube-api-access-grmv5\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc\" (UID: \"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" Nov 03 15:58:21.896298 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.896262 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:58:21.946033 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.945998 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:58:21.946033 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.945999 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:58:21.946289 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.946165 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:58:21.949562 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.949539 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc"] Nov 03 15:58:21.951115 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.951090 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc\" (UID: \"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" Nov 03 15:58:21.951230 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.951124 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc\" (UID: \"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" Nov 03 15:58:21.951230 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.951170 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grmv5\" (UniqueName: \"kubernetes.io/projected/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-kube-api-access-grmv5\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc\" (UID: \"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" Nov 03 15:58:21.951836 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.951817 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc\" (UID: \"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" Nov 03 15:58:21.953662 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:21.953644 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc\" (UID: \"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" Nov 03 15:58:22.012960 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:22.012923 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grmv5\" (UniqueName: \"kubernetes.io/projected/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-kube-api-access-grmv5\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc\" (UID: \"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" Nov 03 15:58:22.156223 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:22.156182 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" Nov 03 15:58:22.546688 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:22.546657 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc"] Nov 03 15:58:22.549801 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:58:22.548881 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4fec9c_c8e2_47d7_bd96_4c35554a3a78.slice/crio-d08c78b5db4c6a87d1e6822f04cdf897ef6d4085736856db0501d0ff30779969 WatchSource:0}: Error finding container d08c78b5db4c6a87d1e6822f04cdf897ef6d4085736856db0501d0ff30779969: Status 404 returned error can't find the container with id d08c78b5db4c6a87d1e6822f04cdf897ef6d4085736856db0501d0ff30779969 Nov 03 15:58:23.079468 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:23.079431 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" event={"ID":"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78","Type":"ContainerStarted","Data":"1b814a270d1720afc2f2a0dfcce3daea9798c4d1ea6d10d3223d62db488cb4f2"} Nov 03 15:58:23.079893 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:23.079482 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" event={"ID":"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78","Type":"ContainerStarted","Data":"d08c78b5db4c6a87d1e6822f04cdf897ef6d4085736856db0501d0ff30779969"} Nov 03 15:58:23.199869 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:23.199803 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" podStartSLOduration=2.1997863 podStartE2EDuration="2.1997863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:58:23.197012113 +0000 UTC m=+3010.627969920" watchObservedRunningTime="2026-11-03 15:58:23.1997863 +0000 UTC m=+3010.630744113" Nov 03 15:58:24.088678 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:24.088636 4290 generic.go:356] "Generic (PLEG): container finished" podID="0e4fec9c-c8e2-47d7-bd96-4c35554a3a78" containerID="1b814a270d1720afc2f2a0dfcce3daea9798c4d1ea6d10d3223d62db488cb4f2" exitCode=0 Nov 03 15:58:24.089141 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:24.088735 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" event={"ID":"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78","Type":"ContainerDied","Data":"1b814a270d1720afc2f2a0dfcce3daea9798c4d1ea6d10d3223d62db488cb4f2"} Nov 03 15:58:25.341647 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:25.338594 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" Nov 03 15:58:25.389350 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:25.389312 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-kube-api-access-grmv5\" (UniqueName: \"kubernetes.io/projected/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-kube-api-access-grmv5\") pod \"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78\" (UID: \"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78\") " Nov 03 15:58:25.389350 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:25.389354 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-maas-api-xks-ca\") pod \"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78\" (UID: \"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78\") " Nov 03 15:58:25.389632 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:25.389519 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-maas-api-openshift-ca\") pod \"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78\" (UID: \"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78\") " Nov 03 15:58:25.390096 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:25.390067 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-maas-api-openshift-ca" pod "0e4fec9c-c8e2-47d7-bd96-4c35554a3a78" (UID: "0e4fec9c-c8e2-47d7-bd96-4c35554a3a78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:58:25.391818 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:25.391793 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-maas-api-xks-ca" pod "0e4fec9c-c8e2-47d7-bd96-4c35554a3a78" (UID: "0e4fec9c-c8e2-47d7-bd96-4c35554a3a78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:58:25.391998 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:25.391980 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-kube-api-access-grmv5" pod "0e4fec9c-c8e2-47d7-bd96-4c35554a3a78" (UID: "0e4fec9c-c8e2-47d7-bd96-4c35554a3a78"). InnerVolumeSpecName "kube-api-access-grmv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:58:25.490765 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:25.490721 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:58:25.490765 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:25.490761 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grmv5\" (UniqueName: \"kubernetes.io/projected/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-kube-api-access-grmv5\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:58:25.490979 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:25.490777 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:58:26.102883 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:26.102848 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" Nov 03 15:58:26.103060 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:26.102847 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc" event={"ID":"0e4fec9c-c8e2-47d7-bd96-4c35554a3a78","Type":"ContainerDied","Data":"d08c78b5db4c6a87d1e6822f04cdf897ef6d4085736856db0501d0ff30779969"} Nov 03 15:58:26.103060 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:26.103018 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d08c78b5db4c6a87d1e6822f04cdf897ef6d4085736856db0501d0ff30779969" Nov 03 15:58:26.257380 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:26.257352 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6"] Nov 03 15:58:27.055011 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:27.054975 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm"] Nov 03 15:58:27.060267 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:27.060240 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kj8lm"] Nov 03 15:58:28.677748 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:28.677675 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b56e54f7-53c1-48f5-a568-fee939df5429" path="/var/lib/kubelet/pods/b56e54f7-53c1-48f5-a568-fee939df5429/volumes" Nov 03 15:58:38.702512 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:58:38.702474 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:58:41.278890 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.278849 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" podUID="3e31080a-88a6-4908-a759-910fb44ac101" containerName="main" containerID="cri-o://6c688352a1901e7205008932e4e03521d1acc863b5040dcb1b44e34dc04b1fb7" gracePeriod=45 Nov 03 15:58:41.549305 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.549184 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:58:41.665315 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.665276 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-tmp-dir\") pod \"3e31080a-88a6-4908-a759-910fb44ac101\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " Nov 03 15:58:41.665503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.665338 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-model-cache\") pod \"3e31080a-88a6-4908-a759-910fb44ac101\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " Nov 03 15:58:41.665503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.665401 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e31080a-88a6-4908-a759-910fb44ac101-tls-certs\" (UniqueName: \"kubernetes.io/secret/3e31080a-88a6-4908-a759-910fb44ac101-tls-certs\") pod \"3e31080a-88a6-4908-a759-910fb44ac101\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " Nov 03 15:58:41.665503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.665442 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-dshm\") pod \"3e31080a-88a6-4908-a759-910fb44ac101\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " Nov 03 15:58:41.665503 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.665478 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-home\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-home\") pod \"3e31080a-88a6-4908-a759-910fb44ac101\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " Nov 03 15:58:41.665791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.665557 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e31080a-88a6-4908-a759-910fb44ac101-kube-api-access-fp9gd\" (UniqueName: \"kubernetes.io/projected/3e31080a-88a6-4908-a759-910fb44ac101-kube-api-access-fp9gd\") pod \"3e31080a-88a6-4908-a759-910fb44ac101\" (UID: \"3e31080a-88a6-4908-a759-910fb44ac101\") " Nov 03 15:58:41.665791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.665573 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-tmp-dir" pod "3e31080a-88a6-4908-a759-910fb44ac101" (UID: "3e31080a-88a6-4908-a759-910fb44ac101"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:58:41.665791 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.665653 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-model-cache" pod "3e31080a-88a6-4908-a759-910fb44ac101" (UID: "3e31080a-88a6-4908-a759-910fb44ac101"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:58:41.665948 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.665819 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-home" pod "3e31080a-88a6-4908-a759-910fb44ac101" (UID: "3e31080a-88a6-4908-a759-910fb44ac101"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:58:41.666017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.665979 4290 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-tmp-dir\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:58:41.666017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.665995 4290 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-model-cache\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:58:41.666017 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.666004 4290 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-home\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:58:41.668047 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.668018 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e31080a-88a6-4908-a759-910fb44ac101-tls-certs" pod "3e31080a-88a6-4908-a759-910fb44ac101" (UID: "3e31080a-88a6-4908-a759-910fb44ac101"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:58:41.668184 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.668048 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e31080a-88a6-4908-a759-910fb44ac101-kube-api-access-fp9gd" pod "3e31080a-88a6-4908-a759-910fb44ac101" (UID: "3e31080a-88a6-4908-a759-910fb44ac101"). InnerVolumeSpecName "kube-api-access-fp9gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:58:41.668184 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.668075 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-dshm" pod "3e31080a-88a6-4908-a759-910fb44ac101" (UID: "3e31080a-88a6-4908-a759-910fb44ac101"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:58:41.767404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.767369 4290 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e31080a-88a6-4908-a759-910fb44ac101-tls-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:58:41.767404 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.767401 4290 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e31080a-88a6-4908-a759-910fb44ac101-dshm\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:58:41.767616 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:41.767414 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp9gd\" (UniqueName: \"kubernetes.io/projected/3e31080a-88a6-4908-a759-910fb44ac101-kube-api-access-fp9gd\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:58:42.210976 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:42.210940 4290 generic.go:356] "Generic (PLEG): container finished" podID="3e31080a-88a6-4908-a759-910fb44ac101" containerID="6c688352a1901e7205008932e4e03521d1acc863b5040dcb1b44e34dc04b1fb7" exitCode=0 Nov 03 15:58:42.211190 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:42.211005 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" Nov 03 15:58:42.211190 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:42.211031 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" event={"ID":"3e31080a-88a6-4908-a759-910fb44ac101","Type":"ContainerDied","Data":"6c688352a1901e7205008932e4e03521d1acc863b5040dcb1b44e34dc04b1fb7"} Nov 03 15:58:42.211190 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:42.211085 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6" event={"ID":"3e31080a-88a6-4908-a759-910fb44ac101","Type":"ContainerDied","Data":"6d3d0784a90bc294a31be092095374f275e2f0176b59581c22b82828d627e519"} Nov 03 15:58:42.211190 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:42.211110 4290 scope.go:122] "RemoveContainer" containerID="6c688352a1901e7205008932e4e03521d1acc863b5040dcb1b44e34dc04b1fb7" Nov 03 15:58:42.227317 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:42.227296 4290 scope.go:122] "RemoveContainer" containerID="6c688352a1901e7205008932e4e03521d1acc863b5040dcb1b44e34dc04b1fb7" Nov 03 15:58:42.227659 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:58:42.227636 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c688352a1901e7205008932e4e03521d1acc863b5040dcb1b44e34dc04b1fb7\": container with ID starting with 6c688352a1901e7205008932e4e03521d1acc863b5040dcb1b44e34dc04b1fb7 not found: ID does not exist" containerID="6c688352a1901e7205008932e4e03521d1acc863b5040dcb1b44e34dc04b1fb7" Nov 03 15:58:42.227751 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:42.227668 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c688352a1901e7205008932e4e03521d1acc863b5040dcb1b44e34dc04b1fb7"} err="failed to get container status \"6c688352a1901e7205008932e4e03521d1acc863b5040dcb1b44e34dc04b1fb7\": rpc error: code = NotFound desc = could not find container \"6c688352a1901e7205008932e4e03521d1acc863b5040dcb1b44e34dc04b1fb7\": container with ID starting with 6c688352a1901e7205008932e4e03521d1acc863b5040dcb1b44e34dc04b1fb7 not found: ID does not exist" Nov 03 15:58:42.379567 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:42.379526 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6"] Nov 03 15:58:42.429228 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:42.429191 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-d9d854/simulator-single-main-d9d854-kserve-79bc75bf88-hdqx6"] Nov 03 15:58:42.677823 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:42.677792 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e31080a-88a6-4908-a759-910fb44ac101" path="/var/lib/kubelet/pods/3e31080a-88a6-4908-a759-910fb44ac101/volumes" Nov 03 15:58:42.878175 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:42.878140 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc"] Nov 03 15:58:42.881229 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:42.881201 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9d854-lj2xc"] Nov 03 15:58:44.676933 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:44.676900 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e4fec9c-c8e2-47d7-bd96-4c35554a3a78" path="/var/lib/kubelet/pods/0e4fec9c-c8e2-47d7-bd96-4c35554a3a78/volumes" Nov 03 15:58:51.935894 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:51.935854 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb"] Nov 03 15:58:51.984156 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:51.984117 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ts7gb"] Nov 03 15:58:52.058435 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.058403 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65"] Nov 03 15:58:52.058921 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.058907 4290 cpu_manager.go:414] "containerMap: removing container" podUID="0e4fec9c-c8e2-47d7-bd96-4c35554a3a78" containerName="revoke-keys" Nov 03 15:58:52.058981 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.058922 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4fec9c-c8e2-47d7-bd96-4c35554a3a78" containerName="revoke-keys" Nov 03 15:58:52.058981 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.058939 4290 cpu_manager.go:414] "containerMap: removing container" podUID="3e31080a-88a6-4908-a759-910fb44ac101" containerName="main" Nov 03 15:58:52.058981 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.058944 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e31080a-88a6-4908-a759-910fb44ac101" containerName="main" Nov 03 15:58:52.059094 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.059034 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4fec9c-c8e2-47d7-bd96-4c35554a3a78" containerName="revoke-keys" Nov 03 15:58:52.059094 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.059044 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e31080a-88a6-4908-a759-910fb44ac101" containerName="main" Nov 03 15:58:52.062251 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.062220 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" Nov 03 15:58:52.125282 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.125248 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:58:52.125475 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.125433 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:58:52.125475 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.125432 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:58:52.141162 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.141125 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:58:52.178124 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.178087 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65"] Nov 03 15:58:52.183274 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.183246 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11ac430-d479-4225-acd2-b6db84781286-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-glj65\" (UID: \"a11ac430-d479-4225-acd2-b6db84781286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" Nov 03 15:58:52.183422 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.183402 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11ac430-d479-4225-acd2-b6db84781286-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-glj65\" (UID: \"a11ac430-d479-4225-acd2-b6db84781286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" Nov 03 15:58:52.183511 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.183495 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dwfv\" (UniqueName: \"kubernetes.io/projected/a11ac430-d479-4225-acd2-b6db84781286-kube-api-access-5dwfv\") pod \"maas-api-revoke-sub-3d4156f1-glj65\" (UID: \"a11ac430-d479-4225-acd2-b6db84781286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" Nov 03 15:58:52.284673 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.284586 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11ac430-d479-4225-acd2-b6db84781286-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-glj65\" (UID: \"a11ac430-d479-4225-acd2-b6db84781286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" Nov 03 15:58:52.284673 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.284655 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dwfv\" (UniqueName: \"kubernetes.io/projected/a11ac430-d479-4225-acd2-b6db84781286-kube-api-access-5dwfv\") pod \"maas-api-revoke-sub-3d4156f1-glj65\" (UID: \"a11ac430-d479-4225-acd2-b6db84781286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" Nov 03 15:58:52.284954 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.284730 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11ac430-d479-4225-acd2-b6db84781286-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-glj65\" (UID: \"a11ac430-d479-4225-acd2-b6db84781286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" Nov 03 15:58:52.285348 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.285314 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11ac430-d479-4225-acd2-b6db84781286-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-glj65\" (UID: \"a11ac430-d479-4225-acd2-b6db84781286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" Nov 03 15:58:52.287311 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.287290 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11ac430-d479-4225-acd2-b6db84781286-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-glj65\" (UID: \"a11ac430-d479-4225-acd2-b6db84781286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" Nov 03 15:58:52.348271 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.348233 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dwfv\" (UniqueName: \"kubernetes.io/projected/a11ac430-d479-4225-acd2-b6db84781286-kube-api-access-5dwfv\") pod \"maas-api-revoke-sub-3d4156f1-glj65\" (UID: \"a11ac430-d479-4225-acd2-b6db84781286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" Nov 03 15:58:52.376525 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.376492 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" Nov 03 15:58:52.687873 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.687835 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393ebecd-95aa-44e3-921d-7825c86b9bec" path="/var/lib/kubelet/pods/393ebecd-95aa-44e3-921d-7825c86b9bec/volumes" Nov 03 15:58:52.765448 ip-10-0-139-35 kubenswrapper[4290]: W1103 15:58:52.765412 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11ac430_d479_4225_acd2_b6db84781286.slice/crio-0b3f2d89fdc72d9aa8794a8fdfbf25de9563faf5b0ba82acf16622de27b73d8b WatchSource:0}: Error finding container 0b3f2d89fdc72d9aa8794a8fdfbf25de9563faf5b0ba82acf16622de27b73d8b: Status 404 returned error can't find the container with id 0b3f2d89fdc72d9aa8794a8fdfbf25de9563faf5b0ba82acf16622de27b73d8b Nov 03 15:58:52.765629 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:52.765505 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65"] Nov 03 15:58:53.289748 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:53.289672 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" event={"ID":"a11ac430-d479-4225-acd2-b6db84781286","Type":"ContainerStarted","Data":"bc5fc61161e2462988248a9ca058283e495bf9b2b75c583ed75c42cfa0e7809f"} Nov 03 15:58:53.290218 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:53.289778 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" event={"ID":"a11ac430-d479-4225-acd2-b6db84781286","Type":"ContainerStarted","Data":"0b3f2d89fdc72d9aa8794a8fdfbf25de9563faf5b0ba82acf16622de27b73d8b"} Nov 03 15:58:53.408396 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:53.408336 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" podStartSLOduration=1.408321572 podStartE2EDuration="1.408321572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:58:53.406126776 +0000 UTC m=+3040.837084585" watchObservedRunningTime="2026-11-03 15:58:53.408321572 +0000 UTC m=+3040.839279407" Nov 03 15:58:54.297814 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:54.297778 4290 generic.go:356] "Generic (PLEG): container finished" podID="a11ac430-d479-4225-acd2-b6db84781286" containerID="bc5fc61161e2462988248a9ca058283e495bf9b2b75c583ed75c42cfa0e7809f" exitCode=0 Nov 03 15:58:54.298237 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:54.297875 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" event={"ID":"a11ac430-d479-4225-acd2-b6db84781286","Type":"ContainerDied","Data":"bc5fc61161e2462988248a9ca058283e495bf9b2b75c583ed75c42cfa0e7809f"} Nov 03 15:58:55.446326 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:55.446298 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" Nov 03 15:58:55.511106 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:55.511067 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11ac430-d479-4225-acd2-b6db84781286-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11ac430-d479-4225-acd2-b6db84781286-maas-api-openshift-ca\") pod \"a11ac430-d479-4225-acd2-b6db84781286\" (UID: \"a11ac430-d479-4225-acd2-b6db84781286\") " Nov 03 15:58:55.511282 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:55.511139 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11ac430-d479-4225-acd2-b6db84781286-kube-api-access-5dwfv\" (UniqueName: \"kubernetes.io/projected/a11ac430-d479-4225-acd2-b6db84781286-kube-api-access-5dwfv\") pod \"a11ac430-d479-4225-acd2-b6db84781286\" (UID: \"a11ac430-d479-4225-acd2-b6db84781286\") " Nov 03 15:58:55.511282 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:55.511246 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a11ac430-d479-4225-acd2-b6db84781286-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11ac430-d479-4225-acd2-b6db84781286-maas-api-xks-ca\") pod \"a11ac430-d479-4225-acd2-b6db84781286\" (UID: \"a11ac430-d479-4225-acd2-b6db84781286\") " Nov 03 15:58:55.511481 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:55.511458 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11ac430-d479-4225-acd2-b6db84781286-maas-api-openshift-ca" pod "a11ac430-d479-4225-acd2-b6db84781286" (UID: "a11ac430-d479-4225-acd2-b6db84781286"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:58:55.511629 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:55.511606 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11ac430-d479-4225-acd2-b6db84781286-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:58:55.513493 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:55.513463 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11ac430-d479-4225-acd2-b6db84781286-kube-api-access-5dwfv" pod "a11ac430-d479-4225-acd2-b6db84781286" (UID: "a11ac430-d479-4225-acd2-b6db84781286"). InnerVolumeSpecName "kube-api-access-5dwfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:58:55.513493 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:55.513473 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11ac430-d479-4225-acd2-b6db84781286-maas-api-xks-ca" pod "a11ac430-d479-4225-acd2-b6db84781286" (UID: "a11ac430-d479-4225-acd2-b6db84781286"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:58:55.612980 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:55.612885 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dwfv\" (UniqueName: \"kubernetes.io/projected/a11ac430-d479-4225-acd2-b6db84781286-kube-api-access-5dwfv\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:58:55.612980 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:55.612924 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11ac430-d479-4225-acd2-b6db84781286-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:58:56.311811 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:56.311778 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" Nov 03 15:58:56.311970 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:56.311780 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65" event={"ID":"a11ac430-d479-4225-acd2-b6db84781286","Type":"ContainerDied","Data":"0b3f2d89fdc72d9aa8794a8fdfbf25de9563faf5b0ba82acf16622de27b73d8b"} Nov 03 15:58:56.311970 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:56.311906 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b3f2d89fdc72d9aa8794a8fdfbf25de9563faf5b0ba82acf16622de27b73d8b" Nov 03 15:58:58.054914 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:58.054881 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25"] Nov 03 15:58:58.107497 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:58.107452 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-28t25"] Nov 03 15:58:58.677080 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:58:58.677044 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35c1568-0a6d-4014-b4c3-a8695a69c325" path="/var/lib/kubelet/pods/d35c1568-0a6d-4014-b4c3-a8695a69c325/volumes" Nov 03 15:59:08.822804 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:59:08.822385 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 15:59:18.387419 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:18.387389 4290 scope.go:122] "RemoveContainer" containerID="b7c04f39e63623cfe953d84bf74ebeea256aa7c008ea8be99cf4e383502f5c02" Nov 03 15:59:22.053712 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:22.053649 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf"] Nov 03 15:59:22.103232 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:22.103199 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2ngxf"] Nov 03 15:59:22.678497 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:22.678460 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac1edee5-bb39-4ce5-8e4e-afeab678bfdf" path="/var/lib/kubelet/pods/ac1edee5-bb39-4ce5-8e4e-afeab678bfdf/volumes" Nov 03 15:59:27.130796 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.130764 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68"] Nov 03 15:59:27.131273 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.131010 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" podUID="e54c8082-6a4c-453d-a283-227ba8f71285" containerName="manager" containerID="cri-o://fbaa85fe0d27db114bfde98f42df5509f44a9fecb3fa80636aea7d1348c1420a" gracePeriod=10 Nov 03 15:59:27.389323 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.389256 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" Nov 03 15:59:27.496789 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.496753 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54c8082-6a4c-453d-a283-227ba8f71285-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e54c8082-6a4c-453d-a283-227ba8f71285-extensions-socket-volume\") pod \"e54c8082-6a4c-453d-a283-227ba8f71285\" (UID: \"e54c8082-6a4c-453d-a283-227ba8f71285\") " Nov 03 15:59:27.496973 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.496947 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54c8082-6a4c-453d-a283-227ba8f71285-kube-api-access-shm8s\" (UniqueName: \"kubernetes.io/projected/e54c8082-6a4c-453d-a283-227ba8f71285-kube-api-access-shm8s\") pod \"e54c8082-6a4c-453d-a283-227ba8f71285\" (UID: \"e54c8082-6a4c-453d-a283-227ba8f71285\") " Nov 03 15:59:27.497196 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.497167 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54c8082-6a4c-453d-a283-227ba8f71285-extensions-socket-volume" pod "e54c8082-6a4c-453d-a283-227ba8f71285" (UID: "e54c8082-6a4c-453d-a283-227ba8f71285"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:59:27.499160 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.499139 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54c8082-6a4c-453d-a283-227ba8f71285-kube-api-access-shm8s" pod "e54c8082-6a4c-453d-a283-227ba8f71285" (UID: "e54c8082-6a4c-453d-a283-227ba8f71285"). InnerVolumeSpecName "kube-api-access-shm8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:59:27.561689 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.561648 4290 generic.go:356] "Generic (PLEG): container finished" podID="e54c8082-6a4c-453d-a283-227ba8f71285" containerID="fbaa85fe0d27db114bfde98f42df5509f44a9fecb3fa80636aea7d1348c1420a" exitCode=0 Nov 03 15:59:27.561911 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.561736 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" Nov 03 15:59:27.561911 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.561764 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" event={"ID":"e54c8082-6a4c-453d-a283-227ba8f71285","Type":"ContainerDied","Data":"fbaa85fe0d27db114bfde98f42df5509f44a9fecb3fa80636aea7d1348c1420a"} Nov 03 15:59:27.561911 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.561836 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68" event={"ID":"e54c8082-6a4c-453d-a283-227ba8f71285","Type":"ContainerDied","Data":"57719c853371d91966ec0212f6253cacef7657e38b1f16ca527d2495c830f68f"} Nov 03 15:59:27.561911 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.561867 4290 scope.go:122] "RemoveContainer" containerID="fbaa85fe0d27db114bfde98f42df5509f44a9fecb3fa80636aea7d1348c1420a" Nov 03 15:59:27.577192 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.577169 4290 scope.go:122] "RemoveContainer" containerID="fbaa85fe0d27db114bfde98f42df5509f44a9fecb3fa80636aea7d1348c1420a" Nov 03 15:59:27.577503 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:59:27.577486 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbaa85fe0d27db114bfde98f42df5509f44a9fecb3fa80636aea7d1348c1420a\": container with ID starting with fbaa85fe0d27db114bfde98f42df5509f44a9fecb3fa80636aea7d1348c1420a not found: ID does not exist" containerID="fbaa85fe0d27db114bfde98f42df5509f44a9fecb3fa80636aea7d1348c1420a" Nov 03 15:59:27.577565 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.577513 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbaa85fe0d27db114bfde98f42df5509f44a9fecb3fa80636aea7d1348c1420a"} err="failed to get container status \"fbaa85fe0d27db114bfde98f42df5509f44a9fecb3fa80636aea7d1348c1420a\": rpc error: code = NotFound desc = could not find container \"fbaa85fe0d27db114bfde98f42df5509f44a9fecb3fa80636aea7d1348c1420a\": container with ID starting with fbaa85fe0d27db114bfde98f42df5509f44a9fecb3fa80636aea7d1348c1420a not found: ID does not exist" Nov 03 15:59:27.598675 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.598648 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shm8s\" (UniqueName: \"kubernetes.io/projected/e54c8082-6a4c-453d-a283-227ba8f71285-kube-api-access-shm8s\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:59:27.598675 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.598674 4290 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e54c8082-6a4c-453d-a283-227ba8f71285-extensions-socket-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:59:27.730943 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.730859 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68"] Nov 03 15:59:27.740510 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:27.740480 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4dh68"] Nov 03 15:59:28.677111 ip-10-0-139-35 kubenswrapper[4290]: I1103 15:59:28.677081 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e54c8082-6a4c-453d-a283-227ba8f71285" path="/var/lib/kubelet/pods/e54c8082-6a4c-453d-a283-227ba8f71285/volumes" Nov 03 15:59:38.701142 ip-10-0-139-35 kubenswrapper[4290]: E1103 15:59:38.701101 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:00:00.178176 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.178096 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz"] Nov 03 16:00:00.178664 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.178648 4290 cpu_manager.go:414] "containerMap: removing container" podUID="e54c8082-6a4c-453d-a283-227ba8f71285" containerName="manager" Nov 03 16:00:00.178730 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.178666 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e54c8082-6a4c-453d-a283-227ba8f71285" containerName="manager" Nov 03 16:00:00.178730 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.178712 4290 cpu_manager.go:414] "containerMap: removing container" podUID="a11ac430-d479-4225-acd2-b6db84781286" containerName="revoke-keys" Nov 03 16:00:00.178730 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.178719 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11ac430-d479-4225-acd2-b6db84781286" containerName="revoke-keys" Nov 03 16:00:00.178837 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.178800 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11ac430-d479-4225-acd2-b6db84781286" containerName="revoke-keys" Nov 03 16:00:00.178837 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.178814 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="e54c8082-6a4c-453d-a283-227ba8f71285" containerName="manager" Nov 03 16:00:00.182187 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.182171 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" Nov 03 16:00:00.237175 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.237141 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:00:00.241728 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.241672 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4jb7\" (UniqueName: \"kubernetes.io/projected/c027a4bf-e7b0-4b19-9fb6-b3acc4edf995-kube-api-access-j4jb7\") pod \"maas-api-key-cleanup-29895360-sskrz\" (UID: \"c027a4bf-e7b0-4b19-9fb6-b3acc4edf995\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" Nov 03 16:00:00.241889 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.241821 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz"] Nov 03 16:00:00.241889 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.241838 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c027a4bf-e7b0-4b19-9fb6-b3acc4edf995-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895360-sskrz\" (UID: \"c027a4bf-e7b0-4b19-9fb6-b3acc4edf995\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" Nov 03 16:00:00.286947 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.286915 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:00:00.287164 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.287043 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 16:00:00.342795 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.342760 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c027a4bf-e7b0-4b19-9fb6-b3acc4edf995-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895360-sskrz\" (UID: \"c027a4bf-e7b0-4b19-9fb6-b3acc4edf995\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" Nov 03 16:00:00.343311 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.342886 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4jb7\" (UniqueName: \"kubernetes.io/projected/c027a4bf-e7b0-4b19-9fb6-b3acc4edf995-kube-api-access-j4jb7\") pod \"maas-api-key-cleanup-29895360-sskrz\" (UID: \"c027a4bf-e7b0-4b19-9fb6-b3acc4edf995\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" Nov 03 16:00:00.343537 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.343514 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c027a4bf-e7b0-4b19-9fb6-b3acc4edf995-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895360-sskrz\" (UID: \"c027a4bf-e7b0-4b19-9fb6-b3acc4edf995\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" Nov 03 16:00:00.405017 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.404984 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4jb7\" (UniqueName: \"kubernetes.io/projected/c027a4bf-e7b0-4b19-9fb6-b3acc4edf995-kube-api-access-j4jb7\") pod \"maas-api-key-cleanup-29895360-sskrz\" (UID: \"c027a4bf-e7b0-4b19-9fb6-b3acc4edf995\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" Nov 03 16:00:00.495728 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.495612 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" Nov 03 16:00:00.878146 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:00.878118 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz"] Nov 03 16:00:00.879305 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:00:00.879282 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc027a4bf_e7b0_4b19_9fb6_b3acc4edf995.slice/crio-55945c26dfec8bd12f9fd72a91861047936e9ae4690de22752d5b0d18c8529fe WatchSource:0}: Error finding container 55945c26dfec8bd12f9fd72a91861047936e9ae4690de22752d5b0d18c8529fe: Status 404 returned error can't find the container with id 55945c26dfec8bd12f9fd72a91861047936e9ae4690de22752d5b0d18c8529fe Nov 03 16:00:01.790978 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:01.790941 4290 generic.go:356] "Generic (PLEG): container finished" podID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" containerID="ec4ca59e1090e09f15007ce61d6ac8bba5d699b187c8ac0fb62ea73479e91a9e" exitCode=60 Nov 03 16:00:01.791402 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:01.790992 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" event={"ID":"c027a4bf-e7b0-4b19-9fb6-b3acc4edf995","Type":"ContainerDied","Data":"ec4ca59e1090e09f15007ce61d6ac8bba5d699b187c8ac0fb62ea73479e91a9e"} Nov 03 16:00:01.791402 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:01.791040 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" event={"ID":"c027a4bf-e7b0-4b19-9fb6-b3acc4edf995","Type":"ContainerStarted","Data":"55945c26dfec8bd12f9fd72a91861047936e9ae4690de22752d5b0d18c8529fe"} Nov 03 16:00:01.791402 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:01.791305 4290 scope.go:122] "RemoveContainer" containerID="ec4ca59e1090e09f15007ce61d6ac8bba5d699b187c8ac0fb62ea73479e91a9e" Nov 03 16:00:02.799723 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:02.799624 4290 generic.go:356] "Generic (PLEG): container finished" podID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" containerID="72b7d2bf6ba714402b927a852bf157a00ecc63a0ae4c325f463aceeb033878cd" exitCode=60 Nov 03 16:00:02.800151 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:02.799724 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" event={"ID":"c027a4bf-e7b0-4b19-9fb6-b3acc4edf995","Type":"ContainerDied","Data":"72b7d2bf6ba714402b927a852bf157a00ecc63a0ae4c325f463aceeb033878cd"} Nov 03 16:00:02.800151 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:02.799789 4290 scope.go:122] "RemoveContainer" containerID="ec4ca59e1090e09f15007ce61d6ac8bba5d699b187c8ac0fb62ea73479e91a9e" Nov 03 16:00:02.800151 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:02.799939 4290 scope.go:122] "RemoveContainer" containerID="72b7d2bf6ba714402b927a852bf157a00ecc63a0ae4c325f463aceeb033878cd" Nov 03 16:00:02.800339 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:00:02.800210 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895360-sskrz_odh-ai-gateway-infra(c027a4bf-e7b0-4b19-9fb6-b3acc4edf995)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" podUID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" Nov 03 16:00:03.807837 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:03.807809 4290 scope.go:122] "RemoveContainer" containerID="72b7d2bf6ba714402b927a852bf157a00ecc63a0ae4c325f463aceeb033878cd" Nov 03 16:00:03.808255 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:00:03.808048 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895360-sskrz_odh-ai-gateway-infra(c027a4bf-e7b0-4b19-9fb6-b3acc4edf995)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" podUID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" Nov 03 16:00:08.704279 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:00:08.704181 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:00:14.671319 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:14.671282 4290 scope.go:122] "RemoveContainer" containerID="72b7d2bf6ba714402b927a852bf157a00ecc63a0ae4c325f463aceeb033878cd" Nov 03 16:00:15.890352 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:15.890320 4290 generic.go:356] "Generic (PLEG): container finished" podID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" containerID="609ddf40cebacae889615fc4172c86aad049ac6e3cfcad0141d00505e851f4c5" exitCode=60 Nov 03 16:00:15.890851 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:15.890394 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" event={"ID":"c027a4bf-e7b0-4b19-9fb6-b3acc4edf995","Type":"ContainerDied","Data":"609ddf40cebacae889615fc4172c86aad049ac6e3cfcad0141d00505e851f4c5"} Nov 03 16:00:15.890851 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:15.890454 4290 scope.go:122] "RemoveContainer" containerID="72b7d2bf6ba714402b927a852bf157a00ecc63a0ae4c325f463aceeb033878cd" Nov 03 16:00:15.890851 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:15.890816 4290 scope.go:122] "RemoveContainer" containerID="609ddf40cebacae889615fc4172c86aad049ac6e3cfcad0141d00505e851f4c5" Nov 03 16:00:15.891102 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:00:15.891083 4290 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895360-sskrz_odh-ai-gateway-infra(c027a4bf-e7b0-4b19-9fb6-b3acc4edf995)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" podUID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" Nov 03 16:00:17.045239 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:17.045197 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz"] Nov 03 16:00:17.199466 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:17.199438 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" Nov 03 16:00:17.319947 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:17.319260 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c027a4bf-e7b0-4b19-9fb6-b3acc4edf995-kube-api-access-j4jb7\" (UniqueName: \"kubernetes.io/projected/c027a4bf-e7b0-4b19-9fb6-b3acc4edf995-kube-api-access-j4jb7\") pod \"c027a4bf-e7b0-4b19-9fb6-b3acc4edf995\" (UID: \"c027a4bf-e7b0-4b19-9fb6-b3acc4edf995\") " Nov 03 16:00:17.319947 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:17.319379 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c027a4bf-e7b0-4b19-9fb6-b3acc4edf995-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c027a4bf-e7b0-4b19-9fb6-b3acc4edf995-maas-api-service-ca\") pod \"c027a4bf-e7b0-4b19-9fb6-b3acc4edf995\" (UID: \"c027a4bf-e7b0-4b19-9fb6-b3acc4edf995\") " Nov 03 16:00:17.319947 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:17.319790 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c027a4bf-e7b0-4b19-9fb6-b3acc4edf995-maas-api-service-ca" pod "c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" (UID: "c027a4bf-e7b0-4b19-9fb6-b3acc4edf995"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:00:17.319947 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:17.319872 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c027a4bf-e7b0-4b19-9fb6-b3acc4edf995-maas-api-service-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:00:17.321958 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:17.321930 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c027a4bf-e7b0-4b19-9fb6-b3acc4edf995-kube-api-access-j4jb7" pod "c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" (UID: "c027a4bf-e7b0-4b19-9fb6-b3acc4edf995"). InnerVolumeSpecName "kube-api-access-j4jb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:00:17.420833 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:17.420799 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4jb7\" (UniqueName: \"kubernetes.io/projected/c027a4bf-e7b0-4b19-9fb6-b3acc4edf995-kube-api-access-j4jb7\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:00:17.906756 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:17.906724 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" Nov 03 16:00:17.906756 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:17.906734 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz" event={"ID":"c027a4bf-e7b0-4b19-9fb6-b3acc4edf995","Type":"ContainerDied","Data":"55945c26dfec8bd12f9fd72a91861047936e9ae4690de22752d5b0d18c8529fe"} Nov 03 16:00:17.907013 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:17.906802 4290 scope.go:122] "RemoveContainer" containerID="609ddf40cebacae889615fc4172c86aad049ac6e3cfcad0141d00505e851f4c5" Nov 03 16:00:18.075072 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:18.075041 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz"] Nov 03 16:00:18.087327 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:18.087298 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895360-sskrz"] Nov 03 16:00:18.418355 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:18.418324 4290 scope.go:122] "RemoveContainer" containerID="d8e0177342e33d0f4d248ef39bb1dfe1f17cb958831ae70338977e83cb470e11" Nov 03 16:00:18.432327 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:18.432307 4290 scope.go:122] "RemoveContainer" containerID="00d08d019c3341b4929e109dff837c16d57e3a2121b5d203645b60b54b47358b" Nov 03 16:00:18.678010 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:18.677920 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" path="/var/lib/kubelet/pods/c027a4bf-e7b0-4b19-9fb6-b3acc4edf995/volumes" Nov 03 16:00:30.770584 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:30.770544 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4"] Nov 03 16:00:30.771134 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:30.771116 4290 cpu_manager.go:414] "containerMap: removing container" podUID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" containerName="cleanup" Nov 03 16:00:30.771196 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:30.771138 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" containerName="cleanup" Nov 03 16:00:30.771196 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:30.771169 4290 cpu_manager.go:414] "containerMap: removing container" podUID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" containerName="cleanup" Nov 03 16:00:30.771196 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:30.771178 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" containerName="cleanup" Nov 03 16:00:30.771312 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:30.771302 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" containerName="cleanup" Nov 03 16:00:30.771349 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:30.771314 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" containerName="cleanup" Nov 03 16:00:30.776260 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:30.776198 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4" Nov 03 16:00:30.833942 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:30.833908 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4"] Nov 03 16:00:30.849832 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:30.849800 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdfe745b-0aad-43f7-aa54-1c44d45bde19-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fzcg4\" (UID: \"cdfe745b-0aad-43f7-aa54-1c44d45bde19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4" Nov 03 16:00:30.850013 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:30.849877 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtrxc\" (UniqueName: \"kubernetes.io/projected/cdfe745b-0aad-43f7-aa54-1c44d45bde19-kube-api-access-rtrxc\") pod \"kuadrant-operator-controller-manager-678d87766f-fzcg4\" (UID: \"cdfe745b-0aad-43f7-aa54-1c44d45bde19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4" Nov 03 16:00:30.879759 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:30.879728 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7ddcq\"" Nov 03 16:00:30.951160 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:30.951116 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtrxc\" (UniqueName: \"kubernetes.io/projected/cdfe745b-0aad-43f7-aa54-1c44d45bde19-kube-api-access-rtrxc\") pod \"kuadrant-operator-controller-manager-678d87766f-fzcg4\" (UID: \"cdfe745b-0aad-43f7-aa54-1c44d45bde19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4" Nov 03 16:00:30.951331 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:30.951218 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdfe745b-0aad-43f7-aa54-1c44d45bde19-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fzcg4\" (UID: \"cdfe745b-0aad-43f7-aa54-1c44d45bde19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4" Nov 03 16:00:30.951552 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:30.951533 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdfe745b-0aad-43f7-aa54-1c44d45bde19-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fzcg4\" (UID: \"cdfe745b-0aad-43f7-aa54-1c44d45bde19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4" Nov 03 16:00:31.018394 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:31.018357 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtrxc\" (UniqueName: \"kubernetes.io/projected/cdfe745b-0aad-43f7-aa54-1c44d45bde19-kube-api-access-rtrxc\") pod \"kuadrant-operator-controller-manager-678d87766f-fzcg4\" (UID: \"cdfe745b-0aad-43f7-aa54-1c44d45bde19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4" Nov 03 16:00:31.089694 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:31.089603 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4" Nov 03 16:00:31.522990 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:31.522959 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4"] Nov 03 16:00:31.524544 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:00:31.524519 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfe745b_0aad_43f7_aa54_1c44d45bde19.slice/crio-fff153f3b98007bc379b8e594ef4a4a7ddfe305ad16a264c35867f2fdae76279 WatchSource:0}: Error finding container fff153f3b98007bc379b8e594ef4a4a7ddfe305ad16a264c35867f2fdae76279: Status 404 returned error can't find the container with id fff153f3b98007bc379b8e594ef4a4a7ddfe305ad16a264c35867f2fdae76279 Nov 03 16:00:31.999557 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:31.999513 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4" event={"ID":"cdfe745b-0aad-43f7-aa54-1c44d45bde19","Type":"ContainerStarted","Data":"1134a28478baeab60e16fcb10e5b81e41350ecbbc6de1c07fb70f541ada2d4ba"} Nov 03 16:00:31.999969 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:31.999566 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4" event={"ID":"cdfe745b-0aad-43f7-aa54-1c44d45bde19","Type":"ContainerStarted","Data":"fff153f3b98007bc379b8e594ef4a4a7ddfe305ad16a264c35867f2fdae76279"} Nov 03 16:00:31.999969 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:31.999598 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4" Nov 03 16:00:32.143653 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:32.143599 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4" podStartSLOduration=2.143586215 podStartE2EDuration="2.143586215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:00:32.139655219 +0000 UTC m=+3139.570613025" watchObservedRunningTime="2026-11-03 16:00:32.143586215 +0000 UTC m=+3139.574544020" Nov 03 16:00:38.701473 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:00:38.701438 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:00:43.009237 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:43.009201 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fzcg4" Nov 03 16:00:55.113372 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:55.113325 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl"] Nov 03 16:00:55.189843 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:55.182602 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-l5txl"] Nov 03 16:00:55.573920 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:55.573801 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pvhdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:00:55.573920 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:55.573891 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pvhdp" podUID="c56971b8-db79-464a-bdaa-1a1e950f7c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:00:55.778866 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:55.778823 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-wl2z4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:00:55.779069 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:55.778921 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" podUID="a3cec29f-05df-404d-b346-53dda11adf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:00:56.677925 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:56.677891 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec3f9fe-f02b-4b99-83fb-42f2353bfc88" path="/var/lib/kubelet/pods/9ec3f9fe-f02b-4b99-83fb-42f2353bfc88/volumes" Nov 03 16:00:57.194135 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.194099 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8"] Nov 03 16:00:57.194629 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.194616 4290 cpu_manager.go:414] "containerMap: removing container" podUID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" containerName="cleanup" Nov 03 16:00:57.194674 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.194631 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" containerName="cleanup" Nov 03 16:00:57.194755 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.194710 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="c027a4bf-e7b0-4b19-9fb6-b3acc4edf995" containerName="cleanup" Nov 03 16:00:57.198024 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.198008 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" Nov 03 16:00:57.257425 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.257398 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8"] Nov 03 16:00:57.302875 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.302844 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:00:57.302875 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.302854 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:00:57.302875 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.302876 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 16:00:57.303122 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.302907 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:00:57.335853 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.335827 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxhtn\" (UniqueName: \"kubernetes.io/projected/25302ba6-6679-42bf-8bb8-03d8a53f268c-kube-api-access-bxhtn\") pod \"maas-api-revoke-sub-d576e68b-p7gt8\" (UID: \"25302ba6-6679-42bf-8bb8-03d8a53f268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" Nov 03 16:00:57.336022 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.335887 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25302ba6-6679-42bf-8bb8-03d8a53f268c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-p7gt8\" (UID: \"25302ba6-6679-42bf-8bb8-03d8a53f268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" Nov 03 16:00:57.336091 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.336020 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25302ba6-6679-42bf-8bb8-03d8a53f268c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-p7gt8\" (UID: \"25302ba6-6679-42bf-8bb8-03d8a53f268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" Nov 03 16:00:57.436651 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.436613 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25302ba6-6679-42bf-8bb8-03d8a53f268c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-p7gt8\" (UID: \"25302ba6-6679-42bf-8bb8-03d8a53f268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" Nov 03 16:00:57.436862 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.436682 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25302ba6-6679-42bf-8bb8-03d8a53f268c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-p7gt8\" (UID: \"25302ba6-6679-42bf-8bb8-03d8a53f268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" Nov 03 16:00:57.436862 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.436815 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxhtn\" (UniqueName: \"kubernetes.io/projected/25302ba6-6679-42bf-8bb8-03d8a53f268c-kube-api-access-bxhtn\") pod \"maas-api-revoke-sub-d576e68b-p7gt8\" (UID: \"25302ba6-6679-42bf-8bb8-03d8a53f268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" Nov 03 16:00:57.437407 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.437383 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25302ba6-6679-42bf-8bb8-03d8a53f268c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-p7gt8\" (UID: \"25302ba6-6679-42bf-8bb8-03d8a53f268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" Nov 03 16:00:57.439251 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.439225 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25302ba6-6679-42bf-8bb8-03d8a53f268c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-p7gt8\" (UID: \"25302ba6-6679-42bf-8bb8-03d8a53f268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" Nov 03 16:00:57.499246 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.499177 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxhtn\" (UniqueName: \"kubernetes.io/projected/25302ba6-6679-42bf-8bb8-03d8a53f268c-kube-api-access-bxhtn\") pod \"maas-api-revoke-sub-d576e68b-p7gt8\" (UID: \"25302ba6-6679-42bf-8bb8-03d8a53f268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" Nov 03 16:00:57.511307 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.511270 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" Nov 03 16:00:57.908272 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:57.908241 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8"] Nov 03 16:00:57.908813 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:00:57.908779 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25302ba6_6679_42bf_8bb8_03d8a53f268c.slice/crio-7c8f0ff39b083f82b7113817047ce1a630b13422920e05930b5d620a332b0c2a WatchSource:0}: Error finding container 7c8f0ff39b083f82b7113817047ce1a630b13422920e05930b5d620a332b0c2a: Status 404 returned error can't find the container with id 7c8f0ff39b083f82b7113817047ce1a630b13422920e05930b5d620a332b0c2a Nov 03 16:00:58.211466 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:58.211383 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" event={"ID":"25302ba6-6679-42bf-8bb8-03d8a53f268c","Type":"ContainerStarted","Data":"7c8f0ff39b083f82b7113817047ce1a630b13422920e05930b5d620a332b0c2a"} Nov 03 16:00:59.220301 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:59.220265 4290 generic.go:356] "Generic (PLEG): container finished" podID="25302ba6-6679-42bf-8bb8-03d8a53f268c" containerID="eec56270a09cb9d3e467c4b97fae36398a0810cd06d08ed45e1571ae3d25cc9f" exitCode=0 Nov 03 16:00:59.220730 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:00:59.220360 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" event={"ID":"25302ba6-6679-42bf-8bb8-03d8a53f268c","Type":"ContainerDied","Data":"eec56270a09cb9d3e467c4b97fae36398a0810cd06d08ed45e1571ae3d25cc9f"} Nov 03 16:01:00.379057 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:00.379034 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" Nov 03 16:01:00.470674 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:00.470639 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25302ba6-6679-42bf-8bb8-03d8a53f268c-kube-api-access-bxhtn\" (UniqueName: \"kubernetes.io/projected/25302ba6-6679-42bf-8bb8-03d8a53f268c-kube-api-access-bxhtn\") pod \"25302ba6-6679-42bf-8bb8-03d8a53f268c\" (UID: \"25302ba6-6679-42bf-8bb8-03d8a53f268c\") " Nov 03 16:01:00.470890 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:00.470743 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25302ba6-6679-42bf-8bb8-03d8a53f268c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25302ba6-6679-42bf-8bb8-03d8a53f268c-maas-api-xks-ca\") pod \"25302ba6-6679-42bf-8bb8-03d8a53f268c\" (UID: \"25302ba6-6679-42bf-8bb8-03d8a53f268c\") " Nov 03 16:01:00.470890 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:00.470782 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25302ba6-6679-42bf-8bb8-03d8a53f268c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25302ba6-6679-42bf-8bb8-03d8a53f268c-maas-api-openshift-ca\") pod \"25302ba6-6679-42bf-8bb8-03d8a53f268c\" (UID: \"25302ba6-6679-42bf-8bb8-03d8a53f268c\") " Nov 03 16:01:00.471263 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:00.471230 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25302ba6-6679-42bf-8bb8-03d8a53f268c-maas-api-openshift-ca" pod "25302ba6-6679-42bf-8bb8-03d8a53f268c" (UID: "25302ba6-6679-42bf-8bb8-03d8a53f268c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:01:00.473056 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:00.473022 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25302ba6-6679-42bf-8bb8-03d8a53f268c-maas-api-xks-ca" pod "25302ba6-6679-42bf-8bb8-03d8a53f268c" (UID: "25302ba6-6679-42bf-8bb8-03d8a53f268c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:01:00.473171 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:00.473106 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25302ba6-6679-42bf-8bb8-03d8a53f268c-kube-api-access-bxhtn" pod "25302ba6-6679-42bf-8bb8-03d8a53f268c" (UID: "25302ba6-6679-42bf-8bb8-03d8a53f268c"). InnerVolumeSpecName "kube-api-access-bxhtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:01:00.572625 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:00.572588 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxhtn\" (UniqueName: \"kubernetes.io/projected/25302ba6-6679-42bf-8bb8-03d8a53f268c-kube-api-access-bxhtn\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:01:00.572824 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:00.572630 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25302ba6-6679-42bf-8bb8-03d8a53f268c-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:01:00.572824 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:00.572646 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25302ba6-6679-42bf-8bb8-03d8a53f268c-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:01:01.234789 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:01.234759 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" Nov 03 16:01:01.234996 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:01.234759 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8" event={"ID":"25302ba6-6679-42bf-8bb8-03d8a53f268c","Type":"ContainerDied","Data":"7c8f0ff39b083f82b7113817047ce1a630b13422920e05930b5d620a332b0c2a"} Nov 03 16:01:01.234996 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:01.234877 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c8f0ff39b083f82b7113817047ce1a630b13422920e05930b5d620a332b0c2a" Nov 03 16:01:06.050199 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:06.050162 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps"] Nov 03 16:01:06.100102 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:06.100070 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-cfqps"] Nov 03 16:01:06.677774 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:06.677735 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6a20c5-0cde-44ee-9047-a85e2a036a6a" path="/var/lib/kubelet/pods/dd6a20c5-0cde-44ee-9047-a85e2a036a6a/volumes" Nov 03 16:01:08.704087 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:01:08.703993 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:01:18.470543 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:18.470505 4290 scope.go:122] "RemoveContainer" containerID="91c1a0d1b3ec2b9ca513a303a62d3a0ef0f479ff28d486675eb599a34b785fae" Nov 03 16:01:18.483131 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:18.483105 4290 scope.go:122] "RemoveContainer" containerID="e2e06ff09bf8171e0e5fb9a5a3c82386a67e1100c11d87934c88abe60072414d" Nov 03 16:01:18.492504 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:18.492481 4290 scope.go:122] "RemoveContainer" containerID="4cd3a2f43d99ff3426ad96fe4e9c8e8a3112d7cd591acb47d175c7e346a8491f" Nov 03 16:01:26.286219 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.286133 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd"] Nov 03 16:01:26.286688 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.286671 4290 cpu_manager.go:414] "containerMap: removing container" podUID="25302ba6-6679-42bf-8bb8-03d8a53f268c" containerName="revoke-keys" Nov 03 16:01:26.286757 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.286691 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25302ba6-6679-42bf-8bb8-03d8a53f268c" containerName="revoke-keys" Nov 03 16:01:26.286829 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.286819 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="25302ba6-6679-42bf-8bb8-03d8a53f268c" containerName="revoke-keys" Nov 03 16:01:26.291461 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.291445 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" Nov 03 16:01:26.348022 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.347990 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd"] Nov 03 16:01:26.378502 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.378472 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-q48kd\" (UID: \"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" Nov 03 16:01:26.378664 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.378555 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkn4x\" (UniqueName: \"kubernetes.io/projected/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-kube-api-access-gkn4x\") pod \"maas-api-revoke-sub-d2bc93bc-q48kd\" (UID: \"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" Nov 03 16:01:26.378664 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.378604 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-q48kd\" (UID: \"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" Nov 03 16:01:26.393603 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.393571 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:01:26.393797 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.393658 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:01:26.393797 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.393693 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 16:01:26.394073 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.394057 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:01:26.479197 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.479168 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-q48kd\" (UID: \"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" Nov 03 16:01:26.479396 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.479254 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-q48kd\" (UID: \"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" Nov 03 16:01:26.479396 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.479297 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkn4x\" (UniqueName: \"kubernetes.io/projected/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-kube-api-access-gkn4x\") pod \"maas-api-revoke-sub-d2bc93bc-q48kd\" (UID: \"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" Nov 03 16:01:26.480024 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.479999 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-q48kd\" (UID: \"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" Nov 03 16:01:26.481766 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.481742 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-q48kd\" (UID: \"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" Nov 03 16:01:26.541909 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.541842 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkn4x\" (UniqueName: \"kubernetes.io/projected/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-kube-api-access-gkn4x\") pod \"maas-api-revoke-sub-d2bc93bc-q48kd\" (UID: \"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" Nov 03 16:01:26.604660 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.604633 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" Nov 03 16:01:26.984248 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:26.984219 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd"] Nov 03 16:01:26.985535 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:01:26.985508 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc68a38b_0fb9_44ba_81c9_f3e34280a9ef.slice/crio-db9fedb2bda6c00f64590e9aedc057ecbd8bc46a076f96ac5311c9503484bcf8 WatchSource:0}: Error finding container db9fedb2bda6c00f64590e9aedc057ecbd8bc46a076f96ac5311c9503484bcf8: Status 404 returned error can't find the container with id db9fedb2bda6c00f64590e9aedc057ecbd8bc46a076f96ac5311c9503484bcf8 Nov 03 16:01:27.405232 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:27.405186 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" event={"ID":"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef","Type":"ContainerStarted","Data":"1c6620a8f8734c4d1a40925d1312977e2faa6c16b94bcb093c519f9367696516"} Nov 03 16:01:27.405646 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:27.405245 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" event={"ID":"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef","Type":"ContainerStarted","Data":"db9fedb2bda6c00f64590e9aedc057ecbd8bc46a076f96ac5311c9503484bcf8"} Nov 03 16:01:27.525442 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:27.525352 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" podStartSLOduration=1.525334862 podStartE2EDuration="1.525334862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:27.522262979 +0000 UTC m=+3194.953220785" watchObservedRunningTime="2026-11-03 16:01:27.525334862 +0000 UTC m=+3194.956292668" Nov 03 16:01:28.414053 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:28.414019 4290 generic.go:356] "Generic (PLEG): container finished" podID="dc68a38b-0fb9-44ba-81c9-f3e34280a9ef" containerID="1c6620a8f8734c4d1a40925d1312977e2faa6c16b94bcb093c519f9367696516" exitCode=0 Nov 03 16:01:28.414469 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:28.414097 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" event={"ID":"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef","Type":"ContainerDied","Data":"1c6620a8f8734c4d1a40925d1312977e2faa6c16b94bcb093c519f9367696516"} Nov 03 16:01:29.565717 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:29.565665 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" Nov 03 16:01:29.607110 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:29.607061 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-maas-api-xks-ca\") pod \"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef\" (UID: \"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef\") " Nov 03 16:01:29.607293 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:29.607130 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-kube-api-access-gkn4x\" (UniqueName: \"kubernetes.io/projected/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-kube-api-access-gkn4x\") pod \"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef\" (UID: \"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef\") " Nov 03 16:01:29.607293 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:29.607165 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-maas-api-openshift-ca\") pod \"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef\" (UID: \"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef\") " Nov 03 16:01:29.607606 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:29.607578 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-maas-api-openshift-ca" pod "dc68a38b-0fb9-44ba-81c9-f3e34280a9ef" (UID: "dc68a38b-0fb9-44ba-81c9-f3e34280a9ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:01:29.609401 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:29.609370 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-maas-api-xks-ca" pod "dc68a38b-0fb9-44ba-81c9-f3e34280a9ef" (UID: "dc68a38b-0fb9-44ba-81c9-f3e34280a9ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:01:29.609496 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:29.609397 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-kube-api-access-gkn4x" pod "dc68a38b-0fb9-44ba-81c9-f3e34280a9ef" (UID: "dc68a38b-0fb9-44ba-81c9-f3e34280a9ef"). InnerVolumeSpecName "kube-api-access-gkn4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:01:29.708525 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:29.708418 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:01:29.708525 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:29.708456 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkn4x\" (UniqueName: \"kubernetes.io/projected/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-kube-api-access-gkn4x\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:01:29.708525 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:29.708471 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:01:30.429733 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:30.429664 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" Nov 03 16:01:30.429733 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:30.429692 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd" event={"ID":"dc68a38b-0fb9-44ba-81c9-f3e34280a9ef","Type":"ContainerDied","Data":"db9fedb2bda6c00f64590e9aedc057ecbd8bc46a076f96ac5311c9503484bcf8"} Nov 03 16:01:30.429998 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:01:30.429770 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db9fedb2bda6c00f64590e9aedc057ecbd8bc46a076f96ac5311c9503484bcf8" Nov 03 16:01:38.701627 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:01:38.701592 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:02:08.706246 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:02:08.706139 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:02:12.429847 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.429813 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74"] Nov 03 16:02:12.430549 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.430528 4290 cpu_manager.go:414] "containerMap: removing container" podUID="dc68a38b-0fb9-44ba-81c9-f3e34280a9ef" containerName="revoke-keys" Nov 03 16:02:12.430655 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.430551 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc68a38b-0fb9-44ba-81c9-f3e34280a9ef" containerName="revoke-keys" Nov 03 16:02:12.430734 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.430689 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc68a38b-0fb9-44ba-81c9-f3e34280a9ef" containerName="revoke-keys" Nov 03 16:02:12.433994 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.433970 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.452565 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.452535 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-model-cache\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.452688 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.452584 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4dlb\" (UniqueName: \"kubernetes.io/projected/0c3cefab-1484-4db1-a0ce-a830acfaed22-kube-api-access-b4dlb\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.452688 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.452656 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-home\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.452818 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.452746 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-dshm\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.452818 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.452798 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-tmp-dir\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.452899 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.452874 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3cefab-1484-4db1-a0ce-a830acfaed22-tls-certs\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.538725 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.538680 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-15832f\"/\"openshift-service-ca.crt\"" Nov 03 16:02:12.538892 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.538777 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-15832f\"/\"default-dockercfg-zgfq4\"" Nov 03 16:02:12.538892 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.538862 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-15832f\"/\"kube-root-ca.crt\"" Nov 03 16:02:12.538967 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.538954 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-15832f\"/\"simulator-single-main-15832f-kserve-self-signed-certs\"" Nov 03 16:02:12.541926 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.541907 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74"] Nov 03 16:02:12.553533 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.553509 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-dshm\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.553690 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.553592 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-tmp-dir\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.553690 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.553642 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3cefab-1484-4db1-a0ce-a830acfaed22-tls-certs\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.553900 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.553692 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-model-cache\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.553900 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.553744 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4dlb\" (UniqueName: \"kubernetes.io/projected/0c3cefab-1484-4db1-a0ce-a830acfaed22-kube-api-access-b4dlb\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.553900 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.553770 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-home\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.554103 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.554050 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-tmp-dir\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.554163 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.554147 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-home\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.554495 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.554477 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-model-cache\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.556487 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.556461 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-dshm\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.556600 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.556461 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3cefab-1484-4db1-a0ce-a830acfaed22-tls-certs\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.616423 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.616395 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4dlb\" (UniqueName: \"kubernetes.io/projected/0c3cefab-1484-4db1-a0ce-a830acfaed22-kube-api-access-b4dlb\") pod \"simulator-single-main-15832f-kserve-7d8c675c65-r6c74\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:12.798285 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.798206 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-15832f\"/\"default-dockercfg-zgfq4\"" Nov 03 16:02:12.807101 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:12.807075 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:13.188811 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:13.188783 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74"] Nov 03 16:02:13.189736 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:02:13.189688 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c3cefab_1484_4db1_a0ce_a830acfaed22.slice/crio-9262e433d233c308709a9bc36c7de9685d9aee62deea4f8ce766123cf6dd3de8 WatchSource:0}: Error finding container 9262e433d233c308709a9bc36c7de9685d9aee62deea4f8ce766123cf6dd3de8: Status 404 returned error can't find the container with id 9262e433d233c308709a9bc36c7de9685d9aee62deea4f8ce766123cf6dd3de8 Nov 03 16:02:13.712775 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:13.712726 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" event={"ID":"0c3cefab-1484-4db1-a0ce-a830acfaed22","Type":"ContainerStarted","Data":"e4f9acf5f7bd69a011ac226f7c0dac39e0de18ce61ebed5db542c42718ce6afc"} Nov 03 16:02:13.713260 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:13.712784 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" event={"ID":"0c3cefab-1484-4db1-a0ce-a830acfaed22","Type":"ContainerStarted","Data":"9262e433d233c308709a9bc36c7de9685d9aee62deea4f8ce766123cf6dd3de8"} Nov 03 16:02:13.807222 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:13.807183 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:13.830629 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:13.830575 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" podStartSLOduration=1.830559666 podStartE2EDuration="1.830559666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:13.829580178 +0000 UTC m=+3241.260537986" watchObservedRunningTime="2026-11-03 16:02:13.830559666 +0000 UTC m=+3241.261517503" Nov 03 16:02:18.526417 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:18.526383 4290 scope.go:122] "RemoveContainer" containerID="e23ec56fe4c1ce1714be0a50eb3f0e4c413b52ad7f2ce4d51cf609fa20921317" Nov 03 16:02:18.536389 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:18.536366 4290 scope.go:122] "RemoveContainer" containerID="31e87b5ac9dba59438044bbfd5aba786048c9eaa0c62e8bbf20f0d439a70b66a" Nov 03 16:02:22.807585 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:22.807547 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:22.820184 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:22.820156 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:23.787454 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:23.787424 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:02:38.703427 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:02:38.703383 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:02:57.486584 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.486503 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8"] Nov 03 16:02:57.491202 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.491186 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" Nov 03 16:02:57.516167 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.516143 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ppr4\" (UniqueName: \"kubernetes.io/projected/75fb2462-eeeb-443e-88f0-3002e5c84d30-kube-api-access-6ppr4\") pod \"maas-api-revoke-sub-453a3524-x6cp8\" (UID: \"75fb2462-eeeb-443e-88f0-3002e5c84d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" Nov 03 16:02:57.516299 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.516187 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75fb2462-eeeb-443e-88f0-3002e5c84d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-453a3524-x6cp8\" (UID: \"75fb2462-eeeb-443e-88f0-3002e5c84d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" Nov 03 16:02:57.516299 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.516260 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75fb2462-eeeb-443e-88f0-3002e5c84d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-453a3524-x6cp8\" (UID: \"75fb2462-eeeb-443e-88f0-3002e5c84d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" Nov 03 16:02:57.594095 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.594064 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:02:57.594292 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.594181 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 16:02:57.594292 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.594216 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:02:57.594292 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.594281 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:02:57.597634 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.597616 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8"] Nov 03 16:02:57.617322 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.617293 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75fb2462-eeeb-443e-88f0-3002e5c84d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-453a3524-x6cp8\" (UID: \"75fb2462-eeeb-443e-88f0-3002e5c84d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" Nov 03 16:02:57.617460 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.617422 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ppr4\" (UniqueName: \"kubernetes.io/projected/75fb2462-eeeb-443e-88f0-3002e5c84d30-kube-api-access-6ppr4\") pod \"maas-api-revoke-sub-453a3524-x6cp8\" (UID: \"75fb2462-eeeb-443e-88f0-3002e5c84d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" Nov 03 16:02:57.617512 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.617461 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75fb2462-eeeb-443e-88f0-3002e5c84d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-453a3524-x6cp8\" (UID: \"75fb2462-eeeb-443e-88f0-3002e5c84d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" Nov 03 16:02:57.618156 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.618130 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75fb2462-eeeb-443e-88f0-3002e5c84d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-453a3524-x6cp8\" (UID: \"75fb2462-eeeb-443e-88f0-3002e5c84d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" Nov 03 16:02:57.619789 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.619765 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75fb2462-eeeb-443e-88f0-3002e5c84d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-453a3524-x6cp8\" (UID: \"75fb2462-eeeb-443e-88f0-3002e5c84d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" Nov 03 16:02:57.680313 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.680283 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ppr4\" (UniqueName: \"kubernetes.io/projected/75fb2462-eeeb-443e-88f0-3002e5c84d30-kube-api-access-6ppr4\") pod \"maas-api-revoke-sub-453a3524-x6cp8\" (UID: \"75fb2462-eeeb-443e-88f0-3002e5c84d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" Nov 03 16:02:57.804832 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:57.804756 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" Nov 03 16:02:58.189984 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:58.189958 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8"] Nov 03 16:02:58.190667 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:02:58.190637 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75fb2462_eeeb_443e_88f0_3002e5c84d30.slice/crio-1c99a8856fe439f82490360f999c1bfbf2ec14c26784eb006bb49f93cc03119a WatchSource:0}: Error finding container 1c99a8856fe439f82490360f999c1bfbf2ec14c26784eb006bb49f93cc03119a: Status 404 returned error can't find the container with id 1c99a8856fe439f82490360f999c1bfbf2ec14c26784eb006bb49f93cc03119a Nov 03 16:02:59.022407 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:59.022366 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" event={"ID":"75fb2462-eeeb-443e-88f0-3002e5c84d30","Type":"ContainerStarted","Data":"2ca9f413ab8f2a7495bf2e38d115e336f5f6bfa435ec8e3a2b15a2738d74a250"} Nov 03 16:02:59.022888 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:02:59.022470 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" event={"ID":"75fb2462-eeeb-443e-88f0-3002e5c84d30","Type":"ContainerStarted","Data":"1c99a8856fe439f82490360f999c1bfbf2ec14c26784eb006bb49f93cc03119a"} Nov 03 16:03:00.035539 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:00.035498 4290 generic.go:356] "Generic (PLEG): container finished" podID="75fb2462-eeeb-443e-88f0-3002e5c84d30" containerID="2ca9f413ab8f2a7495bf2e38d115e336f5f6bfa435ec8e3a2b15a2738d74a250" exitCode=0 Nov 03 16:03:00.036018 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:00.035608 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" event={"ID":"75fb2462-eeeb-443e-88f0-3002e5c84d30","Type":"ContainerDied","Data":"2ca9f413ab8f2a7495bf2e38d115e336f5f6bfa435ec8e3a2b15a2738d74a250"} Nov 03 16:03:00.182514 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:00.182488 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" Nov 03 16:03:00.241889 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:00.241855 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75fb2462-eeeb-443e-88f0-3002e5c84d30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75fb2462-eeeb-443e-88f0-3002e5c84d30-maas-api-openshift-ca\") pod \"75fb2462-eeeb-443e-88f0-3002e5c84d30\" (UID: \"75fb2462-eeeb-443e-88f0-3002e5c84d30\") " Nov 03 16:03:00.242070 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:00.241958 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75fb2462-eeeb-443e-88f0-3002e5c84d30-kube-api-access-6ppr4\" (UniqueName: \"kubernetes.io/projected/75fb2462-eeeb-443e-88f0-3002e5c84d30-kube-api-access-6ppr4\") pod \"75fb2462-eeeb-443e-88f0-3002e5c84d30\" (UID: \"75fb2462-eeeb-443e-88f0-3002e5c84d30\") " Nov 03 16:03:00.242070 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:00.241989 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75fb2462-eeeb-443e-88f0-3002e5c84d30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75fb2462-eeeb-443e-88f0-3002e5c84d30-maas-api-xks-ca\") pod \"75fb2462-eeeb-443e-88f0-3002e5c84d30\" (UID: \"75fb2462-eeeb-443e-88f0-3002e5c84d30\") " Nov 03 16:03:00.242341 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:00.242310 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75fb2462-eeeb-443e-88f0-3002e5c84d30-maas-api-openshift-ca" pod "75fb2462-eeeb-443e-88f0-3002e5c84d30" (UID: "75fb2462-eeeb-443e-88f0-3002e5c84d30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:03:00.244294 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:00.244262 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75fb2462-eeeb-443e-88f0-3002e5c84d30-maas-api-xks-ca" pod "75fb2462-eeeb-443e-88f0-3002e5c84d30" (UID: "75fb2462-eeeb-443e-88f0-3002e5c84d30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:03:00.244392 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:00.244269 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75fb2462-eeeb-443e-88f0-3002e5c84d30-kube-api-access-6ppr4" pod "75fb2462-eeeb-443e-88f0-3002e5c84d30" (UID: "75fb2462-eeeb-443e-88f0-3002e5c84d30"). InnerVolumeSpecName "kube-api-access-6ppr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:03:00.343474 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:00.343379 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ppr4\" (UniqueName: \"kubernetes.io/projected/75fb2462-eeeb-443e-88f0-3002e5c84d30-kube-api-access-6ppr4\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:03:00.343474 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:00.343424 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75fb2462-eeeb-443e-88f0-3002e5c84d30-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:03:00.343474 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:00.343441 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75fb2462-eeeb-443e-88f0-3002e5c84d30-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:03:01.043591 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:01.043510 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" Nov 03 16:03:01.044095 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:01.043506 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8" event={"ID":"75fb2462-eeeb-443e-88f0-3002e5c84d30","Type":"ContainerDied","Data":"1c99a8856fe439f82490360f999c1bfbf2ec14c26784eb006bb49f93cc03119a"} Nov 03 16:03:01.044095 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:01.043639 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c99a8856fe439f82490360f999c1bfbf2ec14c26784eb006bb49f93cc03119a" Nov 03 16:03:08.702787 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:03:08.702671 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:03:09.121155 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.121070 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b"] Nov 03 16:03:09.121579 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.121567 4290 cpu_manager.go:414] "containerMap: removing container" podUID="75fb2462-eeeb-443e-88f0-3002e5c84d30" containerName="revoke-keys" Nov 03 16:03:09.121619 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.121581 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75fb2462-eeeb-443e-88f0-3002e5c84d30" containerName="revoke-keys" Nov 03 16:03:09.121665 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.121655 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="75fb2462-eeeb-443e-88f0-3002e5c84d30" containerName="revoke-keys" Nov 03 16:03:09.125002 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.124985 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" Nov 03 16:03:09.183245 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.183213 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b"] Nov 03 16:03:09.227921 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.227894 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:03:09.227921 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.227915 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:03:09.228158 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.227958 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:03:09.228158 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.228067 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 16:03:09.240905 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.240870 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b\" (UID: \"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" Nov 03 16:03:09.241057 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.240978 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b\" (UID: \"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" Nov 03 16:03:09.241164 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.241145 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vhr\" (UniqueName: \"kubernetes.io/projected/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-kube-api-access-k5vhr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b\" (UID: \"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" Nov 03 16:03:09.341952 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.341918 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vhr\" (UniqueName: \"kubernetes.io/projected/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-kube-api-access-k5vhr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b\" (UID: \"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" Nov 03 16:03:09.342130 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.341983 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b\" (UID: \"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" Nov 03 16:03:09.342130 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.342017 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b\" (UID: \"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" Nov 03 16:03:09.342653 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.342571 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b\" (UID: \"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" Nov 03 16:03:09.344298 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.344276 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b\" (UID: \"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" Nov 03 16:03:09.404312 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.404285 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vhr\" (UniqueName: \"kubernetes.io/projected/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-kube-api-access-k5vhr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b\" (UID: \"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" Nov 03 16:03:09.438189 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.438155 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" Nov 03 16:03:09.817188 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:09.817162 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b"] Nov 03 16:03:09.818978 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:03:09.818947 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5150fa6_2a6f_4a63_ba2d_27ff7bb8aafd.slice/crio-18f8ebe56b8ad068de5965a29dc4dc616bad033305a4dec46844b583fe6e9646 WatchSource:0}: Error finding container 18f8ebe56b8ad068de5965a29dc4dc616bad033305a4dec46844b583fe6e9646: Status 404 returned error can't find the container with id 18f8ebe56b8ad068de5965a29dc4dc616bad033305a4dec46844b583fe6e9646 Nov 03 16:03:10.101470 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:10.101386 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" event={"ID":"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd","Type":"ContainerStarted","Data":"18f8ebe56b8ad068de5965a29dc4dc616bad033305a4dec46844b583fe6e9646"} Nov 03 16:03:11.109315 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:11.109235 4290 generic.go:356] "Generic (PLEG): container finished" podID="c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd" containerID="be226af41e41c4d44d7b6f4d60c8b5599ae6ce7adb3dd90f2d8f503edb46eced" exitCode=0 Nov 03 16:03:11.109315 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:11.109305 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" event={"ID":"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd","Type":"ContainerDied","Data":"be226af41e41c4d44d7b6f4d60c8b5599ae6ce7adb3dd90f2d8f503edb46eced"} Nov 03 16:03:12.262550 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:12.262526 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" Nov 03 16:03:12.375079 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:12.375051 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-kube-api-access-k5vhr\" (UniqueName: \"kubernetes.io/projected/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-kube-api-access-k5vhr\") pod \"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd\" (UID: \"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd\") " Nov 03 16:03:12.375223 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:12.375209 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-maas-api-openshift-ca\") pod \"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd\" (UID: \"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd\") " Nov 03 16:03:12.375271 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:12.375241 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-maas-api-xks-ca\") pod \"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd\" (UID: \"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd\") " Nov 03 16:03:12.375590 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:12.375556 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-maas-api-openshift-ca" pod "c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd" (UID: "c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:03:12.377311 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:12.377269 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-kube-api-access-k5vhr" pod "c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd" (UID: "c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd"). InnerVolumeSpecName "kube-api-access-k5vhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:03:12.377435 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:12.377353 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-maas-api-xks-ca" pod "c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd" (UID: "c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:03:12.476147 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:12.476117 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:03:12.476147 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:12.476145 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:03:12.476353 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:12.476154 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5vhr\" (UniqueName: \"kubernetes.io/projected/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd-kube-api-access-k5vhr\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:03:13.125121 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:13.125087 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" event={"ID":"c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd","Type":"ContainerDied","Data":"18f8ebe56b8ad068de5965a29dc4dc616bad033305a4dec46844b583fe6e9646"} Nov 03 16:03:13.125121 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:13.125115 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b" Nov 03 16:03:13.125319 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:13.125135 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18f8ebe56b8ad068de5965a29dc4dc616bad033305a4dec46844b583fe6e9646" Nov 03 16:03:13.423799 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:13.423689 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 16:03:13.426456 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:13.426430 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 16:03:13.434947 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:13.434924 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 16:03:13.437882 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:13.437861 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 16:03:13.595237 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:13.595200 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74"] Nov 03 16:03:14.058027 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:14.057991 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs"] Nov 03 16:03:14.062081 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:14.062049 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ca95f6-7f2cs"] Nov 03 16:03:14.678427 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:14.678395 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65c66347-024c-4bf6-a7cf-dafe50931382" path="/var/lib/kubelet/pods/65c66347-024c-4bf6-a7cf-dafe50931382/volumes" Nov 03 16:03:18.565320 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:18.565280 4290 scope.go:122] "RemoveContainer" containerID="9ba664ed46c2e3eb7e4ea9a432217175b261837630d1e4f4caac8d3b88bc9db8" Nov 03 16:03:28.616352 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.616305 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" podUID="0c3cefab-1484-4db1-a0ce-a830acfaed22" containerName="main" containerID="cri-o://e4f9acf5f7bd69a011ac226f7c0dac39e0de18ce61ebed5db542c42718ce6afc" gracePeriod=45 Nov 03 16:03:28.777370 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.777329 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" podUID="0c3cefab-1484-4db1-a0ce-a830acfaed22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.197:8000/ready\": dial tcp 10.134.0.197:8000: connect: connection refused" Nov 03 16:03:28.889789 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.889753 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:03:28.950110 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.950076 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-home\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-home\") pod \"0c3cefab-1484-4db1-a0ce-a830acfaed22\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " Nov 03 16:03:28.950295 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.950150 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-model-cache\") pod \"0c3cefab-1484-4db1-a0ce-a830acfaed22\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " Nov 03 16:03:28.950295 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.950173 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-dshm\") pod \"0c3cefab-1484-4db1-a0ce-a830acfaed22\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " Nov 03 16:03:28.950295 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.950221 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c3cefab-1484-4db1-a0ce-a830acfaed22-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3cefab-1484-4db1-a0ce-a830acfaed22-tls-certs\") pod \"0c3cefab-1484-4db1-a0ce-a830acfaed22\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " Nov 03 16:03:28.950295 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.950294 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-tmp-dir\") pod \"0c3cefab-1484-4db1-a0ce-a830acfaed22\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " Nov 03 16:03:28.950527 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.950320 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c3cefab-1484-4db1-a0ce-a830acfaed22-kube-api-access-b4dlb\" (UniqueName: \"kubernetes.io/projected/0c3cefab-1484-4db1-a0ce-a830acfaed22-kube-api-access-b4dlb\") pod \"0c3cefab-1484-4db1-a0ce-a830acfaed22\" (UID: \"0c3cefab-1484-4db1-a0ce-a830acfaed22\") " Nov 03 16:03:28.950527 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.950367 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-home" pod "0c3cefab-1484-4db1-a0ce-a830acfaed22" (UID: "0c3cefab-1484-4db1-a0ce-a830acfaed22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:03:28.950527 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.950390 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-model-cache" pod "0c3cefab-1484-4db1-a0ce-a830acfaed22" (UID: "0c3cefab-1484-4db1-a0ce-a830acfaed22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:03:28.950676 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.950603 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-tmp-dir" pod "0c3cefab-1484-4db1-a0ce-a830acfaed22" (UID: "0c3cefab-1484-4db1-a0ce-a830acfaed22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:03:28.950797 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.950746 4290 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-tmp-dir\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:03:28.950797 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.950762 4290 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-home\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:03:28.950797 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.950771 4290 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-model-cache\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:03:28.952622 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.952591 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-dshm" pod "0c3cefab-1484-4db1-a0ce-a830acfaed22" (UID: "0c3cefab-1484-4db1-a0ce-a830acfaed22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:03:28.952770 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.952712 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c3cefab-1484-4db1-a0ce-a830acfaed22-tls-certs" pod "0c3cefab-1484-4db1-a0ce-a830acfaed22" (UID: "0c3cefab-1484-4db1-a0ce-a830acfaed22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:03:28.952770 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:28.952715 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3cefab-1484-4db1-a0ce-a830acfaed22-kube-api-access-b4dlb" pod "0c3cefab-1484-4db1-a0ce-a830acfaed22" (UID: "0c3cefab-1484-4db1-a0ce-a830acfaed22"). InnerVolumeSpecName "kube-api-access-b4dlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:03:29.051460 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:29.051424 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4dlb\" (UniqueName: \"kubernetes.io/projected/0c3cefab-1484-4db1-a0ce-a830acfaed22-kube-api-access-b4dlb\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:03:29.051460 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:29.051459 4290 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3cefab-1484-4db1-a0ce-a830acfaed22-dshm\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:03:29.051670 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:29.051473 4290 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3cefab-1484-4db1-a0ce-a830acfaed22-tls-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:03:29.232245 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:29.232213 4290 generic.go:356] "Generic (PLEG): container finished" podID="0c3cefab-1484-4db1-a0ce-a830acfaed22" containerID="e4f9acf5f7bd69a011ac226f7c0dac39e0de18ce61ebed5db542c42718ce6afc" exitCode=0 Nov 03 16:03:29.232245 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:29.232250 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" event={"ID":"0c3cefab-1484-4db1-a0ce-a830acfaed22","Type":"ContainerDied","Data":"e4f9acf5f7bd69a011ac226f7c0dac39e0de18ce61ebed5db542c42718ce6afc"} Nov 03 16:03:29.232531 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:29.232279 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" event={"ID":"0c3cefab-1484-4db1-a0ce-a830acfaed22","Type":"ContainerDied","Data":"9262e433d233c308709a9bc36c7de9685d9aee62deea4f8ce766123cf6dd3de8"} Nov 03 16:03:29.232531 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:29.232274 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74" Nov 03 16:03:29.232531 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:29.232291 4290 scope.go:122] "RemoveContainer" containerID="e4f9acf5f7bd69a011ac226f7c0dac39e0de18ce61ebed5db542c42718ce6afc" Nov 03 16:03:29.244690 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:29.244666 4290 scope.go:122] "RemoveContainer" containerID="e4f9acf5f7bd69a011ac226f7c0dac39e0de18ce61ebed5db542c42718ce6afc" Nov 03 16:03:29.245020 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:03:29.245000 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4f9acf5f7bd69a011ac226f7c0dac39e0de18ce61ebed5db542c42718ce6afc\": container with ID starting with e4f9acf5f7bd69a011ac226f7c0dac39e0de18ce61ebed5db542c42718ce6afc not found: ID does not exist" containerID="e4f9acf5f7bd69a011ac226f7c0dac39e0de18ce61ebed5db542c42718ce6afc" Nov 03 16:03:29.245099 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:29.245028 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4f9acf5f7bd69a011ac226f7c0dac39e0de18ce61ebed5db542c42718ce6afc"} err="failed to get container status \"e4f9acf5f7bd69a011ac226f7c0dac39e0de18ce61ebed5db542c42718ce6afc\": rpc error: code = NotFound desc = could not find container \"e4f9acf5f7bd69a011ac226f7c0dac39e0de18ce61ebed5db542c42718ce6afc\": container with ID starting with e4f9acf5f7bd69a011ac226f7c0dac39e0de18ce61ebed5db542c42718ce6afc not found: ID does not exist" Nov 03 16:03:29.398569 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:29.398521 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74"] Nov 03 16:03:29.448350 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:29.448314 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-15832f/simulator-single-main-15832f-kserve-7d8c675c65-r6c74"] Nov 03 16:03:30.513967 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:30.513931 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b"] Nov 03 16:03:30.564038 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:30.564003 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-15832f-hxg4b"] Nov 03 16:03:30.677739 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:30.677678 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c3cefab-1484-4db1-a0ce-a830acfaed22" path="/var/lib/kubelet/pods/0c3cefab-1484-4db1-a0ce-a830acfaed22/volumes" Nov 03 16:03:30.678127 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:30.678113 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd" path="/var/lib/kubelet/pods/c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd/volumes" Nov 03 16:03:38.702648 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:03:38.702606 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:03:50.663981 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.663948 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d"] Nov 03 16:03:50.664442 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.664414 4290 cpu_manager.go:414] "containerMap: removing container" podUID="c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd" containerName="revoke-keys" Nov 03 16:03:50.664442 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.664425 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd" containerName="revoke-keys" Nov 03 16:03:50.664442 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.664442 4290 cpu_manager.go:414] "containerMap: removing container" podUID="0c3cefab-1484-4db1-a0ce-a830acfaed22" containerName="main" Nov 03 16:03:50.664549 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.664448 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c3cefab-1484-4db1-a0ce-a830acfaed22" containerName="main" Nov 03 16:03:50.664549 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.664527 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c3cefab-1484-4db1-a0ce-a830acfaed22" containerName="main" Nov 03 16:03:50.664549 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.664544 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5150fa6-2a6f-4a63-ba2d-27ff7bb8aafd" containerName="revoke-keys" Nov 03 16:03:50.667690 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.667675 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.727091 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.727046 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d"] Nov 03 16:03:50.772772 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.772737 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-30b78fff-kserve-self-signed-certs\"" Nov 03 16:03:50.784204 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.784172 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-dshm\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.784334 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.784247 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-home\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.784384 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.784347 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-tmp-dir\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.784429 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.784400 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-model-cache\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.784491 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.784472 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfq4c\" (UniqueName: \"kubernetes.io/projected/14a11e4f-bfe4-489b-bc7a-2985cee678e6-kube-api-access-sfq4c\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.784538 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.784516 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a11e4f-bfe4-489b-bc7a-2985cee678e6-tls-certs\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.885872 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.885836 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-home\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.886050 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.885886 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-tmp-dir\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.886050 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.886006 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-model-cache\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.886171 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.886057 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfq4c\" (UniqueName: \"kubernetes.io/projected/14a11e4f-bfe4-489b-bc7a-2985cee678e6-kube-api-access-sfq4c\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.886171 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.886100 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a11e4f-bfe4-489b-bc7a-2985cee678e6-tls-certs\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.886293 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.886202 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-dshm\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.886360 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.886302 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-tmp-dir\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.886360 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.886345 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-home\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.886468 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.886401 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-model-cache\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.888645 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.888621 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-dshm\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.888870 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.888851 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a11e4f-bfe4-489b-bc7a-2985cee678e6-tls-certs\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.947432 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.947360 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfq4c\" (UniqueName: \"kubernetes.io/projected/14a11e4f-bfe4-489b-bc7a-2985cee678e6-kube-api-access-sfq4c\") pod \"e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:50.979963 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:50.979930 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:51.376277 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:51.376248 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d"] Nov 03 16:03:51.377218 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:03:51.377183 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14a11e4f_bfe4_489b_bc7a_2985cee678e6.slice/crio-804f1f0ac079da0f68559ddfcfe5e96d5c342534e4dc6d58bae70d294c436515 WatchSource:0}: Error finding container 804f1f0ac079da0f68559ddfcfe5e96d5c342534e4dc6d58bae70d294c436515: Status 404 returned error can't find the container with id 804f1f0ac079da0f68559ddfcfe5e96d5c342534e4dc6d58bae70d294c436515 Nov 03 16:03:52.391274 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:52.391236 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" event={"ID":"14a11e4f-bfe4-489b-bc7a-2985cee678e6","Type":"ContainerStarted","Data":"5f613a26d2c2ba16e117af087bbba245dd3a0e28bfdf7016d9c0ea9db9e3e09c"} Nov 03 16:03:52.391274 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:52.391277 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" event={"ID":"14a11e4f-bfe4-489b-bc7a-2985cee678e6","Type":"ContainerStarted","Data":"804f1f0ac079da0f68559ddfcfe5e96d5c342534e4dc6d58bae70d294c436515"} Nov 03 16:03:52.511443 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:52.511377 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" podStartSLOduration=2.511362172 podStartE2EDuration="2.511362172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:03:52.50959604 +0000 UTC m=+3339.940553846" watchObservedRunningTime="2026-11-03 16:03:52.511362172 +0000 UTC m=+3339.942320019" Nov 03 16:03:52.980241 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:52.980212 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:03:56.056728 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:56.054439 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65"] Nov 03 16:03:56.103530 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:56.103492 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glj65"] Nov 03 16:03:56.681377 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:03:56.681335 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11ac430-d479-4225-acd2-b6db84781286" path="/var/lib/kubelet/pods/a11ac430-d479-4225-acd2-b6db84781286/volumes" Nov 03 16:04:00.980395 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:00.980353 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:04:00.993083 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:00.993052 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:04:01.466952 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:01.466924 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:04:08.705374 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:04:08.705297 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:04:14.331810 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.331772 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz"] Nov 03 16:04:14.336649 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.336624 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.395134 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.395100 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz"] Nov 03 16:04:14.440868 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.440835 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-30b78fff-kserve-self-signed-certs\"" Nov 03 16:04:14.476395 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.476355 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ggx\" (UniqueName: \"kubernetes.io/projected/2b45d000-65a7-4c6b-95bf-9af485eb53e0-kube-api-access-l2ggx\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.476595 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.476410 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-home\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.476595 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.476455 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-model-cache\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.476595 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.476517 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-tmp-dir\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.476595 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.476550 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-dshm\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.476595 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.476591 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b45d000-65a7-4c6b-95bf-9af485eb53e0-tls-certs\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.577413 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.577376 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-home\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.577621 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.577424 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-model-cache\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.577621 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.577478 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-tmp-dir\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.577621 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.577507 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-dshm\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.577842 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.577665 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b45d000-65a7-4c6b-95bf-9af485eb53e0-tls-certs\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.577842 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.577808 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ggx\" (UniqueName: \"kubernetes.io/projected/2b45d000-65a7-4c6b-95bf-9af485eb53e0-kube-api-access-l2ggx\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.577951 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.577930 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-home\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.578039 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.577998 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-tmp-dir\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.578039 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.578019 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-model-cache\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.580105 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.580085 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-dshm\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.580400 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.580381 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b45d000-65a7-4c6b-95bf-9af485eb53e0-tls-certs\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.640307 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.640269 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ggx\" (UniqueName: \"kubernetes.io/projected/2b45d000-65a7-4c6b-95bf-9af485eb53e0-kube-api-access-l2ggx\") pod \"e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:14.648539 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:14.648504 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:15.037002 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:15.036969 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz"] Nov 03 16:04:15.552233 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:15.552196 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" event={"ID":"2b45d000-65a7-4c6b-95bf-9af485eb53e0","Type":"ContainerStarted","Data":"fb44d9e4ca3a08c114d214fd8d3ca895dc5a398434fd3e08f0cb219355f95b8d"} Nov 03 16:04:15.552233 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:15.552235 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" event={"ID":"2b45d000-65a7-4c6b-95bf-9af485eb53e0","Type":"ContainerStarted","Data":"9480ed9438e5a4e1c182199a766b57ab44ec24d08c264e96c8267a01650c5eca"} Nov 03 16:04:15.649231 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:15.649200 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:15.675921 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:15.675873 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" podStartSLOduration=1.675858459 podStartE2EDuration="1.675858459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:04:15.671550707 +0000 UTC m=+3363.102508512" watchObservedRunningTime="2026-11-03 16:04:15.675858459 +0000 UTC m=+3363.106816259" Nov 03 16:04:18.601221 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:18.601186 4290 scope.go:122] "RemoveContainer" containerID="5ff12f9eaf1a38d2540d8463f6e584c6444e95e2ceb9e0ddbc8e1de849dd27c6" Nov 03 16:04:24.648720 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:24.648646 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:24.661900 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:24.661869 4290 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:25.655566 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:25.655538 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:04:38.702052 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:04:38.702009 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:04:45.791414 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:45.791373 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k"] Nov 03 16:04:45.798500 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:45.798475 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" Nov 03 16:04:45.851644 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:45.851605 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k"] Nov 03 16:04:45.861368 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:45.861329 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlf7b\" (UniqueName: \"kubernetes.io/projected/2055023e-998a-4aba-a4b5-942fcec069da-kube-api-access-zlf7b\") pod \"maas-api-revoke-sub-452b6d7b-7p25k\" (UID: \"2055023e-998a-4aba-a4b5-942fcec069da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" Nov 03 16:04:45.861525 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:45.861423 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2055023e-998a-4aba-a4b5-942fcec069da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-7p25k\" (UID: \"2055023e-998a-4aba-a4b5-942fcec069da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" Nov 03 16:04:45.861580 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:45.861550 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2055023e-998a-4aba-a4b5-942fcec069da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-7p25k\" (UID: \"2055023e-998a-4aba-a4b5-942fcec069da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" Nov 03 16:04:45.896969 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:45.896936 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:04:45.896969 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:45.896936 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:04:45.897192 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:45.896988 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 16:04:45.903414 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:45.903393 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:04:45.962313 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:45.962276 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2055023e-998a-4aba-a4b5-942fcec069da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-7p25k\" (UID: \"2055023e-998a-4aba-a4b5-942fcec069da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" Nov 03 16:04:45.962492 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:45.962370 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2055023e-998a-4aba-a4b5-942fcec069da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-7p25k\" (UID: \"2055023e-998a-4aba-a4b5-942fcec069da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" Nov 03 16:04:45.962492 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:45.962417 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlf7b\" (UniqueName: \"kubernetes.io/projected/2055023e-998a-4aba-a4b5-942fcec069da-kube-api-access-zlf7b\") pod \"maas-api-revoke-sub-452b6d7b-7p25k\" (UID: \"2055023e-998a-4aba-a4b5-942fcec069da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" Nov 03 16:04:45.963117 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:45.963093 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2055023e-998a-4aba-a4b5-942fcec069da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-7p25k\" (UID: \"2055023e-998a-4aba-a4b5-942fcec069da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" Nov 03 16:04:45.964793 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:45.964776 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2055023e-998a-4aba-a4b5-942fcec069da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-7p25k\" (UID: \"2055023e-998a-4aba-a4b5-942fcec069da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" Nov 03 16:04:46.024682 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:46.024650 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlf7b\" (UniqueName: \"kubernetes.io/projected/2055023e-998a-4aba-a4b5-942fcec069da-kube-api-access-zlf7b\") pod \"maas-api-revoke-sub-452b6d7b-7p25k\" (UID: \"2055023e-998a-4aba-a4b5-942fcec069da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" Nov 03 16:04:46.112523 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:46.112444 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" Nov 03 16:04:46.493899 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:46.493869 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k"] Nov 03 16:04:46.495763 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:04:46.495732 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2055023e_998a_4aba_a4b5_942fcec069da.slice/crio-288cec8eff6dd9cef5ae483745d509ca71b42b39153a626d4d98e710433cd988 WatchSource:0}: Error finding container 288cec8eff6dd9cef5ae483745d509ca71b42b39153a626d4d98e710433cd988: Status 404 returned error can't find the container with id 288cec8eff6dd9cef5ae483745d509ca71b42b39153a626d4d98e710433cd988 Nov 03 16:04:46.798732 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:46.798688 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" event={"ID":"2055023e-998a-4aba-a4b5-942fcec069da","Type":"ContainerStarted","Data":"6f27a129cbd9dbb18d11c3e0c158ade74dd068b146760f9282296fc0216b3192"} Nov 03 16:04:46.799126 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:46.798754 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" event={"ID":"2055023e-998a-4aba-a4b5-942fcec069da","Type":"ContainerStarted","Data":"288cec8eff6dd9cef5ae483745d509ca71b42b39153a626d4d98e710433cd988"} Nov 03 16:04:46.916891 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:46.916844 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" podStartSLOduration=1.916830562 podStartE2EDuration="1.916830562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:04:46.914904888 +0000 UTC m=+3394.345862692" watchObservedRunningTime="2026-11-03 16:04:46.916830562 +0000 UTC m=+3394.347788397" Nov 03 16:04:47.809512 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:47.809410 4290 generic.go:356] "Generic (PLEG): container finished" podID="2055023e-998a-4aba-a4b5-942fcec069da" containerID="6f27a129cbd9dbb18d11c3e0c158ade74dd068b146760f9282296fc0216b3192" exitCode=0 Nov 03 16:04:47.810063 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:47.809505 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" event={"ID":"2055023e-998a-4aba-a4b5-942fcec069da","Type":"ContainerDied","Data":"6f27a129cbd9dbb18d11c3e0c158ade74dd068b146760f9282296fc0216b3192"} Nov 03 16:04:49.277735 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:49.275907 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" Nov 03 16:04:49.430259 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:49.430216 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2055023e-998a-4aba-a4b5-942fcec069da-kube-api-access-zlf7b\" (UniqueName: \"kubernetes.io/projected/2055023e-998a-4aba-a4b5-942fcec069da-kube-api-access-zlf7b\") pod \"2055023e-998a-4aba-a4b5-942fcec069da\" (UID: \"2055023e-998a-4aba-a4b5-942fcec069da\") " Nov 03 16:04:49.430259 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:49.430267 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2055023e-998a-4aba-a4b5-942fcec069da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2055023e-998a-4aba-a4b5-942fcec069da-maas-api-openshift-ca\") pod \"2055023e-998a-4aba-a4b5-942fcec069da\" (UID: \"2055023e-998a-4aba-a4b5-942fcec069da\") " Nov 03 16:04:49.430547 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:49.430291 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2055023e-998a-4aba-a4b5-942fcec069da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2055023e-998a-4aba-a4b5-942fcec069da-maas-api-xks-ca\") pod \"2055023e-998a-4aba-a4b5-942fcec069da\" (UID: \"2055023e-998a-4aba-a4b5-942fcec069da\") " Nov 03 16:04:49.430616 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:49.430588 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2055023e-998a-4aba-a4b5-942fcec069da-maas-api-openshift-ca" pod "2055023e-998a-4aba-a4b5-942fcec069da" (UID: "2055023e-998a-4aba-a4b5-942fcec069da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:04:49.430891 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:49.430871 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2055023e-998a-4aba-a4b5-942fcec069da-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:04:49.432645 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:49.432625 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2055023e-998a-4aba-a4b5-942fcec069da-maas-api-xks-ca" pod "2055023e-998a-4aba-a4b5-942fcec069da" (UID: "2055023e-998a-4aba-a4b5-942fcec069da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:04:49.432757 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:49.432682 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2055023e-998a-4aba-a4b5-942fcec069da-kube-api-access-zlf7b" pod "2055023e-998a-4aba-a4b5-942fcec069da" (UID: "2055023e-998a-4aba-a4b5-942fcec069da"). InnerVolumeSpecName "kube-api-access-zlf7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:04:49.532371 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:49.532288 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlf7b\" (UniqueName: \"kubernetes.io/projected/2055023e-998a-4aba-a4b5-942fcec069da-kube-api-access-zlf7b\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:04:49.532371 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:49.532317 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2055023e-998a-4aba-a4b5-942fcec069da-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:04:49.833080 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:49.832995 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" event={"ID":"2055023e-998a-4aba-a4b5-942fcec069da","Type":"ContainerDied","Data":"288cec8eff6dd9cef5ae483745d509ca71b42b39153a626d4d98e710433cd988"} Nov 03 16:04:49.833080 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:49.833017 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k" Nov 03 16:04:49.833080 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:49.833042 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="288cec8eff6dd9cef5ae483745d509ca71b42b39153a626d4d98e710433cd988" Nov 03 16:04:58.850367 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:04:58.850333 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d"] Nov 03 16:05:07.712314 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:07.712272 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz"] Nov 03 16:05:08.708015 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:05:08.707921 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:05:13.872481 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:13.872445 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" podUID="14a11e4f-bfe4-489b-bc7a-2985cee678e6" containerName="main" containerID="cri-o://5f613a26d2c2ba16e117af087bbba245dd3a0e28bfdf7016d9c0ea9db9e3e09c" gracePeriod=45 Nov 03 16:05:14.038021 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.037988 4290 generic.go:356] "Generic (PLEG): container finished" podID="14a11e4f-bfe4-489b-bc7a-2985cee678e6" containerID="5f613a26d2c2ba16e117af087bbba245dd3a0e28bfdf7016d9c0ea9db9e3e09c" exitCode=0 Nov 03 16:05:14.038181 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.038027 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" event={"ID":"14a11e4f-bfe4-489b-bc7a-2985cee678e6","Type":"ContainerDied","Data":"5f613a26d2c2ba16e117af087bbba245dd3a0e28bfdf7016d9c0ea9db9e3e09c"} Nov 03 16:05:14.151469 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.151440 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:05:14.247140 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.247105 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14a11e4f-bfe4-489b-bc7a-2985cee678e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/14a11e4f-bfe4-489b-bc7a-2985cee678e6-tls-certs\") pod \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " Nov 03 16:05:14.247319 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.247267 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-tmp-dir\") pod \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " Nov 03 16:05:14.247319 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.247289 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-model-cache\") pod \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " Nov 03 16:05:14.247415 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.247333 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a11e4f-bfe4-489b-bc7a-2985cee678e6-kube-api-access-sfq4c\" (UniqueName: \"kubernetes.io/projected/14a11e4f-bfe4-489b-bc7a-2985cee678e6-kube-api-access-sfq4c\") pod \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " Nov 03 16:05:14.247415 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.247351 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-home\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-home\") pod \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " Nov 03 16:05:14.247415 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.247380 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-dshm\") pod \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\" (UID: \"14a11e4f-bfe4-489b-bc7a-2985cee678e6\") " Nov 03 16:05:14.247622 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.247586 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-model-cache" pod "14a11e4f-bfe4-489b-bc7a-2985cee678e6" (UID: "14a11e4f-bfe4-489b-bc7a-2985cee678e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:05:14.247727 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.247609 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-tmp-dir" pod "14a11e4f-bfe4-489b-bc7a-2985cee678e6" (UID: "14a11e4f-bfe4-489b-bc7a-2985cee678e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:05:14.247727 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.247601 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-home" pod "14a11e4f-bfe4-489b-bc7a-2985cee678e6" (UID: "14a11e4f-bfe4-489b-bc7a-2985cee678e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:05:14.247854 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.247818 4290 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-tmp-dir\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:14.247854 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.247833 4290 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-model-cache\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:14.247854 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.247847 4290 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-home\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:14.249757 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.249667 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-dshm" pod "14a11e4f-bfe4-489b-bc7a-2985cee678e6" (UID: "14a11e4f-bfe4-489b-bc7a-2985cee678e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:05:14.249757 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.249694 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a11e4f-bfe4-489b-bc7a-2985cee678e6-kube-api-access-sfq4c" pod "14a11e4f-bfe4-489b-bc7a-2985cee678e6" (UID: "14a11e4f-bfe4-489b-bc7a-2985cee678e6"). InnerVolumeSpecName "kube-api-access-sfq4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:05:14.249890 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.249762 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a11e4f-bfe4-489b-bc7a-2985cee678e6-tls-certs" pod "14a11e4f-bfe4-489b-bc7a-2985cee678e6" (UID: "14a11e4f-bfe4-489b-bc7a-2985cee678e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:05:14.349351 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.349307 4290 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a11e4f-bfe4-489b-bc7a-2985cee678e6-tls-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:14.349351 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.349344 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfq4c\" (UniqueName: \"kubernetes.io/projected/14a11e4f-bfe4-489b-bc7a-2985cee678e6-kube-api-access-sfq4c\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:14.349351 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:14.349359 4290 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a11e4f-bfe4-489b-bc7a-2985cee678e6-dshm\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:15.046326 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:15.046246 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" Nov 03 16:05:15.046326 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:15.046273 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d" event={"ID":"14a11e4f-bfe4-489b-bc7a-2985cee678e6","Type":"ContainerDied","Data":"804f1f0ac079da0f68559ddfcfe5e96d5c342534e4dc6d58bae70d294c436515"} Nov 03 16:05:15.046830 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:15.046328 4290 scope.go:122] "RemoveContainer" containerID="5f613a26d2c2ba16e117af087bbba245dd3a0e28bfdf7016d9c0ea9db9e3e09c" Nov 03 16:05:15.212657 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:15.212623 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d"] Nov 03 16:05:15.262666 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:15.262634 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-30b78fff-kserve-6d7dd7f98f-nbs6d"] Nov 03 16:05:16.677693 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:16.677663 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a11e4f-bfe4-489b-bc7a-2985cee678e6" path="/var/lib/kubelet/pods/14a11e4f-bfe4-489b-bc7a-2985cee678e6/volumes" Nov 03 16:05:18.632112 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:18.632072 4290 scope.go:122] "RemoveContainer" containerID="bc5fc61161e2462988248a9ca058283e495bf9b2b75c583ed75c42cfa0e7809f" Nov 03 16:05:18.642002 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:18.641977 4290 scope.go:122] "RemoveContainer" containerID="1b814a270d1720afc2f2a0dfcce3daea9798c4d1ea6d10d3223d62db488cb4f2" Nov 03 16:05:22.733052 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:22.733003 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" podUID="2b45d000-65a7-4c6b-95bf-9af485eb53e0" containerName="main" containerID="cri-o://fb44d9e4ca3a08c114d214fd8d3ca895dc5a398434fd3e08f0cb219355f95b8d" gracePeriod=45 Nov 03 16:05:22.992364 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:22.992305 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:05:23.039604 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.039570 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-tmp-dir\") pod \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " Nov 03 16:05:23.039788 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.039666 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b45d000-65a7-4c6b-95bf-9af485eb53e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/2b45d000-65a7-4c6b-95bf-9af485eb53e0-tls-certs\") pod \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " Nov 03 16:05:23.039872 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.039790 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-model-cache\") pod \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " Nov 03 16:05:23.039872 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.039831 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-home\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-home\") pod \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " Nov 03 16:05:23.039872 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.039857 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-dshm\") pod \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " Nov 03 16:05:23.040052 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.039909 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b45d000-65a7-4c6b-95bf-9af485eb53e0-kube-api-access-l2ggx\" (UniqueName: \"kubernetes.io/projected/2b45d000-65a7-4c6b-95bf-9af485eb53e0-kube-api-access-l2ggx\") pod \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\" (UID: \"2b45d000-65a7-4c6b-95bf-9af485eb53e0\") " Nov 03 16:05:23.040052 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.039917 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-tmp-dir" pod "2b45d000-65a7-4c6b-95bf-9af485eb53e0" (UID: "2b45d000-65a7-4c6b-95bf-9af485eb53e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:05:23.040162 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.040106 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-model-cache" pod "2b45d000-65a7-4c6b-95bf-9af485eb53e0" (UID: "2b45d000-65a7-4c6b-95bf-9af485eb53e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:05:23.040162 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.040114 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-home" pod "2b45d000-65a7-4c6b-95bf-9af485eb53e0" (UID: "2b45d000-65a7-4c6b-95bf-9af485eb53e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:05:23.040358 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.040316 4290 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-model-cache\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:23.040358 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.040339 4290 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-home\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:23.040358 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.040352 4290 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-tmp-dir\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:23.042081 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.042058 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b45d000-65a7-4c6b-95bf-9af485eb53e0-tls-certs" pod "2b45d000-65a7-4c6b-95bf-9af485eb53e0" (UID: "2b45d000-65a7-4c6b-95bf-9af485eb53e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:05:23.042437 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.042422 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-dshm" pod "2b45d000-65a7-4c6b-95bf-9af485eb53e0" (UID: "2b45d000-65a7-4c6b-95bf-9af485eb53e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:05:23.042637 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.042624 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b45d000-65a7-4c6b-95bf-9af485eb53e0-kube-api-access-l2ggx" pod "2b45d000-65a7-4c6b-95bf-9af485eb53e0" (UID: "2b45d000-65a7-4c6b-95bf-9af485eb53e0"). InnerVolumeSpecName "kube-api-access-l2ggx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:05:23.122929 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.122891 4290 generic.go:356] "Generic (PLEG): container finished" podID="2b45d000-65a7-4c6b-95bf-9af485eb53e0" containerID="fb44d9e4ca3a08c114d214fd8d3ca895dc5a398434fd3e08f0cb219355f95b8d" exitCode=0 Nov 03 16:05:23.122929 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.122934 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" Nov 03 16:05:23.123257 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.122969 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" event={"ID":"2b45d000-65a7-4c6b-95bf-9af485eb53e0","Type":"ContainerDied","Data":"fb44d9e4ca3a08c114d214fd8d3ca895dc5a398434fd3e08f0cb219355f95b8d"} Nov 03 16:05:23.123257 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.123014 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz" event={"ID":"2b45d000-65a7-4c6b-95bf-9af485eb53e0","Type":"ContainerDied","Data":"9480ed9438e5a4e1c182199a766b57ab44ec24d08c264e96c8267a01650c5eca"} Nov 03 16:05:23.123257 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.123032 4290 scope.go:122] "RemoveContainer" containerID="fb44d9e4ca3a08c114d214fd8d3ca895dc5a398434fd3e08f0cb219355f95b8d" Nov 03 16:05:23.135715 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.135680 4290 scope.go:122] "RemoveContainer" containerID="fb44d9e4ca3a08c114d214fd8d3ca895dc5a398434fd3e08f0cb219355f95b8d" Nov 03 16:05:23.136022 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:05:23.136002 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb44d9e4ca3a08c114d214fd8d3ca895dc5a398434fd3e08f0cb219355f95b8d\": container with ID starting with fb44d9e4ca3a08c114d214fd8d3ca895dc5a398434fd3e08f0cb219355f95b8d not found: ID does not exist" containerID="fb44d9e4ca3a08c114d214fd8d3ca895dc5a398434fd3e08f0cb219355f95b8d" Nov 03 16:05:23.136103 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.136030 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb44d9e4ca3a08c114d214fd8d3ca895dc5a398434fd3e08f0cb219355f95b8d"} err="failed to get container status \"fb44d9e4ca3a08c114d214fd8d3ca895dc5a398434fd3e08f0cb219355f95b8d\": rpc error: code = NotFound desc = could not find container \"fb44d9e4ca3a08c114d214fd8d3ca895dc5a398434fd3e08f0cb219355f95b8d\": container with ID starting with fb44d9e4ca3a08c114d214fd8d3ca895dc5a398434fd3e08f0cb219355f95b8d not found: ID does not exist" Nov 03 16:05:23.141624 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.141598 4290 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b45d000-65a7-4c6b-95bf-9af485eb53e0-tls-certs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:23.141624 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.141623 4290 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b45d000-65a7-4c6b-95bf-9af485eb53e0-dshm\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:23.141804 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.141633 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2ggx\" (UniqueName: \"kubernetes.io/projected/2b45d000-65a7-4c6b-95bf-9af485eb53e0-kube-api-access-l2ggx\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:23.289220 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.289130 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz"] Nov 03 16:05:23.293160 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:23.293132 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-30b78fff-kserve-545b684b85-rcmsz"] Nov 03 16:05:24.676850 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:24.676818 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b45d000-65a7-4c6b-95bf-9af485eb53e0" path="/var/lib/kubelet/pods/2b45d000-65a7-4c6b-95bf-9af485eb53e0/volumes" Nov 03 16:05:38.705934 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:05:38.705900 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:05:44.721847 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.721815 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x"] Nov 03 16:05:44.722593 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.722571 4290 cpu_manager.go:414] "containerMap: removing container" podUID="14a11e4f-bfe4-489b-bc7a-2985cee678e6" containerName="main" Nov 03 16:05:44.722640 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.722597 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a11e4f-bfe4-489b-bc7a-2985cee678e6" containerName="main" Nov 03 16:05:44.722640 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.722626 4290 cpu_manager.go:414] "containerMap: removing container" podUID="2b45d000-65a7-4c6b-95bf-9af485eb53e0" containerName="main" Nov 03 16:05:44.722640 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.722632 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b45d000-65a7-4c6b-95bf-9af485eb53e0" containerName="main" Nov 03 16:05:44.722762 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.722654 4290 cpu_manager.go:414] "containerMap: removing container" podUID="2055023e-998a-4aba-a4b5-942fcec069da" containerName="revoke-keys" Nov 03 16:05:44.722762 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.722659 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2055023e-998a-4aba-a4b5-942fcec069da" containerName="revoke-keys" Nov 03 16:05:44.722828 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.722785 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a11e4f-bfe4-489b-bc7a-2985cee678e6" containerName="main" Nov 03 16:05:44.722828 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.722805 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b45d000-65a7-4c6b-95bf-9af485eb53e0" containerName="main" Nov 03 16:05:44.722828 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.722819 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="2055023e-998a-4aba-a4b5-942fcec069da" containerName="revoke-keys" Nov 03 16:05:44.727388 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.727371 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" Nov 03 16:05:44.782750 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.782716 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x"] Nov 03 16:05:44.817425 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.817396 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a8d6c0-5t24x\" (UID: \"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" Nov 03 16:05:44.817587 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.817461 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a8d6c0-5t24x\" (UID: \"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" Nov 03 16:05:44.817587 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.817539 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z264q\" (UniqueName: \"kubernetes.io/projected/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-kube-api-access-z264q\") pod \"maas-api-revoke-sub-61a8d6c0-5t24x\" (UID: \"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" Nov 03 16:05:44.828736 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.828692 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:05:44.828892 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.828873 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:05:44.828953 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.828906 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:05:44.831076 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.831051 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 16:05:44.919080 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.919052 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a8d6c0-5t24x\" (UID: \"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" Nov 03 16:05:44.919231 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.919109 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a8d6c0-5t24x\" (UID: \"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" Nov 03 16:05:44.919231 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.919128 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z264q\" (UniqueName: \"kubernetes.io/projected/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-kube-api-access-z264q\") pod \"maas-api-revoke-sub-61a8d6c0-5t24x\" (UID: \"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" Nov 03 16:05:44.919735 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.919675 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a8d6c0-5t24x\" (UID: \"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" Nov 03 16:05:44.921689 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.921664 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a8d6c0-5t24x\" (UID: \"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" Nov 03 16:05:44.981087 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:44.981024 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z264q\" (UniqueName: \"kubernetes.io/projected/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-kube-api-access-z264q\") pod \"maas-api-revoke-sub-61a8d6c0-5t24x\" (UID: \"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" Nov 03 16:05:45.042097 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:45.042061 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" Nov 03 16:05:45.425545 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:45.425516 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x"] Nov 03 16:05:45.426324 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:05:45.426296 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1f5e6c_cf01_4c7e_9b48_73a937fd18dd.slice/crio-d9fbe123ba8a61111080ee929e8559030e68ef8cbabe5e2af9ac2a74fa40a87f WatchSource:0}: Error finding container d9fbe123ba8a61111080ee929e8559030e68ef8cbabe5e2af9ac2a74fa40a87f: Status 404 returned error can't find the container with id d9fbe123ba8a61111080ee929e8559030e68ef8cbabe5e2af9ac2a74fa40a87f Nov 03 16:05:46.293087 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:46.293044 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" event={"ID":"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd","Type":"ContainerStarted","Data":"0d161defa6ea4aa989d6d75ac1d6a1b84ba6820ece14029d3324abd76b92d2c5"} Nov 03 16:05:46.293476 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:46.293110 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" event={"ID":"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd","Type":"ContainerStarted","Data":"d9fbe123ba8a61111080ee929e8559030e68ef8cbabe5e2af9ac2a74fa40a87f"} Nov 03 16:05:46.412562 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:46.412512 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" podStartSLOduration=2.412497859 podStartE2EDuration="2.412497859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:05:46.409850142 +0000 UTC m=+3453.840807960" watchObservedRunningTime="2026-11-03 16:05:46.412497859 +0000 UTC m=+3453.843455665" Nov 03 16:05:47.301172 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:47.301139 4290 generic.go:356] "Generic (PLEG): container finished" podID="dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd" containerID="0d161defa6ea4aa989d6d75ac1d6a1b84ba6820ece14029d3324abd76b92d2c5" exitCode=0 Nov 03 16:05:47.301829 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:47.301174 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" event={"ID":"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd","Type":"ContainerDied","Data":"0d161defa6ea4aa989d6d75ac1d6a1b84ba6820ece14029d3324abd76b92d2c5"} Nov 03 16:05:48.453867 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:48.453836 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" Nov 03 16:05:48.558534 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:48.558441 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-kube-api-access-z264q\" (UniqueName: \"kubernetes.io/projected/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-kube-api-access-z264q\") pod \"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd\" (UID: \"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd\") " Nov 03 16:05:48.558534 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:48.558502 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-maas-api-openshift-ca\") pod \"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd\" (UID: \"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd\") " Nov 03 16:05:48.558834 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:48.558597 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-maas-api-xks-ca\") pod \"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd\" (UID: \"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd\") " Nov 03 16:05:48.558979 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:48.558951 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-maas-api-openshift-ca" pod "dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd" (UID: "dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:05:48.560807 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:48.560784 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-maas-api-xks-ca" pod "dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd" (UID: "dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:05:48.560807 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:48.560792 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-kube-api-access-z264q" pod "dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd" (UID: "dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd"). InnerVolumeSpecName "kube-api-access-z264q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:05:48.659996 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:48.659954 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z264q\" (UniqueName: \"kubernetes.io/projected/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-kube-api-access-z264q\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:48.659996 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:48.659988 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:48.659996 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:48.660004 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:49.315139 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:49.315114 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" Nov 03 16:05:49.315316 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:49.315103 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x" event={"ID":"dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd","Type":"ContainerDied","Data":"d9fbe123ba8a61111080ee929e8559030e68ef8cbabe5e2af9ac2a74fa40a87f"} Nov 03 16:05:49.315316 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:05:49.315247 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9fbe123ba8a61111080ee929e8559030e68ef8cbabe5e2af9ac2a74fa40a87f" Nov 03 16:06:01.050906 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:01.050831 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8"] Nov 03 16:06:01.100314 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:01.100286 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7gt8"] Nov 03 16:06:02.678853 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:02.678820 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25302ba6-6679-42bf-8bb8-03d8a53f268c" path="/var/lib/kubelet/pods/25302ba6-6679-42bf-8bb8-03d8a53f268c/volumes" Nov 03 16:06:08.706774 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:06:08.706666 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:06:12.392803 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:12.392762 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh"] Nov 03 16:06:12.393618 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:12.393595 4290 cpu_manager.go:414] "containerMap: removing container" podUID="dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd" containerName="revoke-keys" Nov 03 16:06:12.393787 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:12.393620 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd" containerName="revoke-keys" Nov 03 16:06:12.393858 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:12.393801 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd" containerName="revoke-keys" Nov 03 16:06:12.397595 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:12.397573 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" Nov 03 16:06:12.456019 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:12.455990 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh"] Nov 03 16:06:12.468510 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:12.468476 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2cca28a2\"" Nov 03 16:06:12.468654 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:12.468597 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2cca28a2-dockercfg-tgwcr\"" Nov 03 16:06:12.518616 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:12.518586 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b5906f-ab56-4a76-8529-719c19a56e39-plugins-config-volume\") pod \"payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh\" (UID: \"23b5906f-ab56-4a76-8529-719c19a56e39\") " pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" Nov 03 16:06:12.518790 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:12.518695 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdb9\" (UniqueName: \"kubernetes.io/projected/23b5906f-ab56-4a76-8529-719c19a56e39-kube-api-access-gjdb9\") pod \"payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh\" (UID: \"23b5906f-ab56-4a76-8529-719c19a56e39\") " pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" Nov 03 16:06:12.620007 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:12.619969 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdb9\" (UniqueName: \"kubernetes.io/projected/23b5906f-ab56-4a76-8529-719c19a56e39-kube-api-access-gjdb9\") pod \"payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh\" (UID: \"23b5906f-ab56-4a76-8529-719c19a56e39\") " pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" Nov 03 16:06:12.620174 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:12.620067 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b5906f-ab56-4a76-8529-719c19a56e39-plugins-config-volume\") pod \"payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh\" (UID: \"23b5906f-ab56-4a76-8529-719c19a56e39\") " pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" Nov 03 16:06:12.620730 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:12.620693 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b5906f-ab56-4a76-8529-719c19a56e39-plugins-config-volume\") pod \"payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh\" (UID: \"23b5906f-ab56-4a76-8529-719c19a56e39\") " pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" Nov 03 16:06:12.682124 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:12.682037 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdb9\" (UniqueName: \"kubernetes.io/projected/23b5906f-ab56-4a76-8529-719c19a56e39-kube-api-access-gjdb9\") pod \"payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh\" (UID: \"23b5906f-ab56-4a76-8529-719c19a56e39\") " pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" Nov 03 16:06:12.714551 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:12.714521 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" Nov 03 16:06:13.096363 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:13.096327 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh"] Nov 03 16:06:13.096824 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:06:13.096799 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b5906f_ab56_4a76_8529_719c19a56e39.slice/crio-50f1b922913eaa0014c2404ef7e47b695d43258d385f305dfe938229438bacd6 WatchSource:0}: Error finding container 50f1b922913eaa0014c2404ef7e47b695d43258d385f305dfe938229438bacd6: Status 404 returned error can't find the container with id 50f1b922913eaa0014c2404ef7e47b695d43258d385f305dfe938229438bacd6 Nov 03 16:06:13.490189 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:13.490153 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" event={"ID":"23b5906f-ab56-4a76-8529-719c19a56e39","Type":"ContainerStarted","Data":"f4ca8b96d944c6f7c8f5062d8555b849741491b53044e236e3a0b6280b03a32b"} Nov 03 16:06:13.490629 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:13.490212 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" event={"ID":"23b5906f-ab56-4a76-8529-719c19a56e39","Type":"ContainerStarted","Data":"50f1b922913eaa0014c2404ef7e47b695d43258d385f305dfe938229438bacd6"} Nov 03 16:06:13.490629 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:13.490348 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" Nov 03 16:06:13.610858 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:13.610805 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" podStartSLOduration=1.610790439 podStartE2EDuration="1.610790439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:06:13.60738498 +0000 UTC m=+3481.038342788" watchObservedRunningTime="2026-11-03 16:06:13.610790439 +0000 UTC m=+3481.041748244" Nov 03 16:06:24.499250 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:24.499222 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" Nov 03 16:06:30.054474 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:30.054440 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd"] Nov 03 16:06:30.104094 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:30.104060 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q48kd"] Nov 03 16:06:30.677677 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:30.677645 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc68a38b-0fb9-44ba-81c9-f3e34280a9ef" path="/var/lib/kubelet/pods/dc68a38b-0fb9-44ba-81c9-f3e34280a9ef/volumes" Nov 03 16:06:34.820182 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:34.820141 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx"] Nov 03 16:06:34.825310 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:34.825290 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" Nov 03 16:06:34.878013 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:34.877979 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:06:34.928057 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:34.928021 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:06:34.928057 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:34.928026 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:06:34.928057 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:34.928033 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 16:06:34.931465 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:34.931437 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx"] Nov 03 16:06:34.958613 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:34.958573 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105634cd-29df-4f32-9ad9-0a4619ee96b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx\" (UID: \"105634cd-29df-4f32-9ad9-0a4619ee96b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" Nov 03 16:06:34.958799 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:34.958734 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgqkv\" (UniqueName: \"kubernetes.io/projected/105634cd-29df-4f32-9ad9-0a4619ee96b6-kube-api-access-qgqkv\") pod \"maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx\" (UID: \"105634cd-29df-4f32-9ad9-0a4619ee96b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" Nov 03 16:06:34.958799 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:34.958760 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105634cd-29df-4f32-9ad9-0a4619ee96b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx\" (UID: \"105634cd-29df-4f32-9ad9-0a4619ee96b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" Nov 03 16:06:35.059562 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:35.059518 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgqkv\" (UniqueName: \"kubernetes.io/projected/105634cd-29df-4f32-9ad9-0a4619ee96b6-kube-api-access-qgqkv\") pod \"maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx\" (UID: \"105634cd-29df-4f32-9ad9-0a4619ee96b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" Nov 03 16:06:35.059562 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:35.059561 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105634cd-29df-4f32-9ad9-0a4619ee96b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx\" (UID: \"105634cd-29df-4f32-9ad9-0a4619ee96b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" Nov 03 16:06:35.059840 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:35.059662 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105634cd-29df-4f32-9ad9-0a4619ee96b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx\" (UID: \"105634cd-29df-4f32-9ad9-0a4619ee96b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" Nov 03 16:06:35.060429 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:35.060408 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105634cd-29df-4f32-9ad9-0a4619ee96b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx\" (UID: \"105634cd-29df-4f32-9ad9-0a4619ee96b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" Nov 03 16:06:35.062271 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:35.062250 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105634cd-29df-4f32-9ad9-0a4619ee96b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx\" (UID: \"105634cd-29df-4f32-9ad9-0a4619ee96b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" Nov 03 16:06:35.121570 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:35.121489 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgqkv\" (UniqueName: \"kubernetes.io/projected/105634cd-29df-4f32-9ad9-0a4619ee96b6-kube-api-access-qgqkv\") pod \"maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx\" (UID: \"105634cd-29df-4f32-9ad9-0a4619ee96b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" Nov 03 16:06:35.138521 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:35.138489 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" Nov 03 16:06:35.518567 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:35.518537 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx"] Nov 03 16:06:35.519456 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:06:35.519431 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105634cd_29df_4f32_9ad9_0a4619ee96b6.slice/crio-671f8ee9a9a79a2394cfd241c59995bb77172866270f784571bbd992ebec877a WatchSource:0}: Error finding container 671f8ee9a9a79a2394cfd241c59995bb77172866270f784571bbd992ebec877a: Status 404 returned error can't find the container with id 671f8ee9a9a79a2394cfd241c59995bb77172866270f784571bbd992ebec877a Nov 03 16:06:35.637627 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:35.637587 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" event={"ID":"105634cd-29df-4f32-9ad9-0a4619ee96b6","Type":"ContainerStarted","Data":"671f8ee9a9a79a2394cfd241c59995bb77172866270f784571bbd992ebec877a"} Nov 03 16:06:36.645549 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:36.645519 4290 generic.go:356] "Generic (PLEG): container finished" podID="105634cd-29df-4f32-9ad9-0a4619ee96b6" containerID="b8d534a887dfb1af873c473b164686d143d5a6d67cb870f1bd6ddcabdaf321a4" exitCode=0 Nov 03 16:06:36.645981 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:36.645580 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" event={"ID":"105634cd-29df-4f32-9ad9-0a4619ee96b6","Type":"ContainerDied","Data":"b8d534a887dfb1af873c473b164686d143d5a6d67cb870f1bd6ddcabdaf321a4"} Nov 03 16:06:37.792145 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:37.792116 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" Nov 03 16:06:37.889132 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:37.889093 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105634cd-29df-4f32-9ad9-0a4619ee96b6-kube-api-access-qgqkv\" (UniqueName: \"kubernetes.io/projected/105634cd-29df-4f32-9ad9-0a4619ee96b6-kube-api-access-qgqkv\") pod \"105634cd-29df-4f32-9ad9-0a4619ee96b6\" (UID: \"105634cd-29df-4f32-9ad9-0a4619ee96b6\") " Nov 03 16:06:37.889330 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:37.889206 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/105634cd-29df-4f32-9ad9-0a4619ee96b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105634cd-29df-4f32-9ad9-0a4619ee96b6-maas-api-xks-ca\") pod \"105634cd-29df-4f32-9ad9-0a4619ee96b6\" (UID: \"105634cd-29df-4f32-9ad9-0a4619ee96b6\") " Nov 03 16:06:37.889330 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:37.889250 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/105634cd-29df-4f32-9ad9-0a4619ee96b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105634cd-29df-4f32-9ad9-0a4619ee96b6-maas-api-openshift-ca\") pod \"105634cd-29df-4f32-9ad9-0a4619ee96b6\" (UID: \"105634cd-29df-4f32-9ad9-0a4619ee96b6\") " Nov 03 16:06:37.889633 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:37.889610 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105634cd-29df-4f32-9ad9-0a4619ee96b6-maas-api-openshift-ca" pod "105634cd-29df-4f32-9ad9-0a4619ee96b6" (UID: "105634cd-29df-4f32-9ad9-0a4619ee96b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:06:37.891380 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:37.891357 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105634cd-29df-4f32-9ad9-0a4619ee96b6-maas-api-xks-ca" pod "105634cd-29df-4f32-9ad9-0a4619ee96b6" (UID: "105634cd-29df-4f32-9ad9-0a4619ee96b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:06:37.891481 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:37.891362 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105634cd-29df-4f32-9ad9-0a4619ee96b6-kube-api-access-qgqkv" pod "105634cd-29df-4f32-9ad9-0a4619ee96b6" (UID: "105634cd-29df-4f32-9ad9-0a4619ee96b6"). InnerVolumeSpecName "kube-api-access-qgqkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:06:37.990422 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:37.990334 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105634cd-29df-4f32-9ad9-0a4619ee96b6-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:06:37.990422 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:37.990370 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105634cd-29df-4f32-9ad9-0a4619ee96b6-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:06:37.990422 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:37.990382 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgqkv\" (UniqueName: \"kubernetes.io/projected/105634cd-29df-4f32-9ad9-0a4619ee96b6-kube-api-access-qgqkv\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:06:38.658931 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:38.658886 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" event={"ID":"105634cd-29df-4f32-9ad9-0a4619ee96b6","Type":"ContainerDied","Data":"671f8ee9a9a79a2394cfd241c59995bb77172866270f784571bbd992ebec877a"} Nov 03 16:06:38.658931 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:38.658919 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx" Nov 03 16:06:38.659154 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:38.658976 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="671f8ee9a9a79a2394cfd241c59995bb77172866270f784571bbd992ebec877a" Nov 03 16:06:38.701400 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:06:38.701361 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:06:46.309994 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:46.309958 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh"] Nov 03 16:06:51.310632 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.310562 4290 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" podUID="23b5906f-ab56-4a76-8529-719c19a56e39" containerName="payload-processing" containerID="cri-o://f4ca8b96d944c6f7c8f5062d8555b849741491b53044e236e3a0b6280b03a32b" gracePeriod=25 Nov 03 16:06:51.571209 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.571155 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" Nov 03 16:06:51.634204 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.634179 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b5906f-ab56-4a76-8529-719c19a56e39-kube-api-access-gjdb9\" (UniqueName: \"kubernetes.io/projected/23b5906f-ab56-4a76-8529-719c19a56e39-kube-api-access-gjdb9\") pod \"23b5906f-ab56-4a76-8529-719c19a56e39\" (UID: \"23b5906f-ab56-4a76-8529-719c19a56e39\") " Nov 03 16:06:51.634332 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.634219 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23b5906f-ab56-4a76-8529-719c19a56e39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b5906f-ab56-4a76-8529-719c19a56e39-plugins-config-volume\") pod \"23b5906f-ab56-4a76-8529-719c19a56e39\" (UID: \"23b5906f-ab56-4a76-8529-719c19a56e39\") " Nov 03 16:06:51.634584 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.634564 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b5906f-ab56-4a76-8529-719c19a56e39-plugins-config-volume" pod "23b5906f-ab56-4a76-8529-719c19a56e39" (UID: "23b5906f-ab56-4a76-8529-719c19a56e39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:06:51.636370 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.636349 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b5906f-ab56-4a76-8529-719c19a56e39-kube-api-access-gjdb9" pod "23b5906f-ab56-4a76-8529-719c19a56e39" (UID: "23b5906f-ab56-4a76-8529-719c19a56e39"). InnerVolumeSpecName "kube-api-access-gjdb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:06:51.735043 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.735010 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjdb9\" (UniqueName: \"kubernetes.io/projected/23b5906f-ab56-4a76-8529-719c19a56e39-kube-api-access-gjdb9\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:06:51.735043 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.735037 4290 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b5906f-ab56-4a76-8529-719c19a56e39-plugins-config-volume\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:06:51.743727 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.743668 4290 generic.go:356] "Generic (PLEG): container finished" podID="23b5906f-ab56-4a76-8529-719c19a56e39" containerID="f4ca8b96d944c6f7c8f5062d8555b849741491b53044e236e3a0b6280b03a32b" exitCode=0 Nov 03 16:06:51.743900 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.743737 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" Nov 03 16:06:51.743900 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.743770 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" event={"ID":"23b5906f-ab56-4a76-8529-719c19a56e39","Type":"ContainerDied","Data":"f4ca8b96d944c6f7c8f5062d8555b849741491b53044e236e3a0b6280b03a32b"} Nov 03 16:06:51.743900 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.743838 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh" event={"ID":"23b5906f-ab56-4a76-8529-719c19a56e39","Type":"ContainerDied","Data":"50f1b922913eaa0014c2404ef7e47b695d43258d385f305dfe938229438bacd6"} Nov 03 16:06:51.743900 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.743869 4290 scope.go:122] "RemoveContainer" containerID="f4ca8b96d944c6f7c8f5062d8555b849741491b53044e236e3a0b6280b03a32b" Nov 03 16:06:51.753932 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.753915 4290 scope.go:122] "RemoveContainer" containerID="f4ca8b96d944c6f7c8f5062d8555b849741491b53044e236e3a0b6280b03a32b" Nov 03 16:06:51.754203 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:06:51.754173 4290 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ca8b96d944c6f7c8f5062d8555b849741491b53044e236e3a0b6280b03a32b\": container with ID starting with f4ca8b96d944c6f7c8f5062d8555b849741491b53044e236e3a0b6280b03a32b not found: ID does not exist" containerID="f4ca8b96d944c6f7c8f5062d8555b849741491b53044e236e3a0b6280b03a32b" Nov 03 16:06:51.754268 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.754207 4290 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ca8b96d944c6f7c8f5062d8555b849741491b53044e236e3a0b6280b03a32b"} err="failed to get container status \"f4ca8b96d944c6f7c8f5062d8555b849741491b53044e236e3a0b6280b03a32b\": rpc error: code = NotFound desc = could not find container \"f4ca8b96d944c6f7c8f5062d8555b849741491b53044e236e3a0b6280b03a32b\": container with ID starting with f4ca8b96d944c6f7c8f5062d8555b849741491b53044e236e3a0b6280b03a32b not found: ID does not exist" Nov 03 16:06:51.911511 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.911477 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh"] Nov 03 16:06:51.914686 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:51.914661 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cca28a2-7795fc47f9-fc6fh"] Nov 03 16:06:52.678672 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:52.678642 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23b5906f-ab56-4a76-8529-719c19a56e39" path="/var/lib/kubelet/pods/23b5906f-ab56-4a76-8529-719c19a56e39/volumes" Nov 03 16:06:55.109932 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:55.109899 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx"] Nov 03 16:06:55.159418 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:55.159385 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cca28a2-8swrx"] Nov 03 16:06:56.678640 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:06:56.678604 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105634cd-29df-4f32-9ad9-0a4619ee96b6" path="/var/lib/kubelet/pods/105634cd-29df-4f32-9ad9-0a4619ee96b6/volumes" Nov 03 16:07:08.701802 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:07:08.701687 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:07:18.706964 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:07:18.706929 4290 scope.go:122] "RemoveContainer" containerID="eec56270a09cb9d3e467c4b97fae36398a0810cd06d08ed45e1571ae3d25cc9f" Nov 03 16:07:38.702982 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:07:38.702944 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:08:01.052147 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:01.052108 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8"] Nov 03 16:08:01.101673 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:01.101636 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-453a3524-x6cp8"] Nov 03 16:08:02.678398 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:02.678363 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75fb2462-eeeb-443e-88f0-3002e5c84d30" path="/var/lib/kubelet/pods/75fb2462-eeeb-443e-88f0-3002e5c84d30/volumes" Nov 03 16:08:08.704776 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:08:08.704652 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:08:13.474693 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:13.474663 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 16:08:13.477779 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:13.477755 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 16:08:13.488084 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:13.488058 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 16:08:13.490832 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:13.490807 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 16:08:18.738211 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:18.738173 4290 scope.go:122] "RemoveContainer" containerID="1c6620a8f8734c4d1a40925d1312977e2faa6c16b94bcb093c519f9367696516" Nov 03 16:08:38.702463 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:08:38.702425 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:08:48.447249 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.447211 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64"] Nov 03 16:08:48.448004 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.447984 4290 cpu_manager.go:414] "containerMap: removing container" podUID="23b5906f-ab56-4a76-8529-719c19a56e39" containerName="payload-processing" Nov 03 16:08:48.448066 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.448006 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b5906f-ab56-4a76-8529-719c19a56e39" containerName="payload-processing" Nov 03 16:08:48.448066 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.448031 4290 cpu_manager.go:414] "containerMap: removing container" podUID="105634cd-29df-4f32-9ad9-0a4619ee96b6" containerName="revoke-keys" Nov 03 16:08:48.448066 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.448036 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105634cd-29df-4f32-9ad9-0a4619ee96b6" containerName="revoke-keys" Nov 03 16:08:48.448175 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.448117 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b5906f-ab56-4a76-8529-719c19a56e39" containerName="payload-processing" Nov 03 16:08:48.448175 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.448128 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="105634cd-29df-4f32-9ad9-0a4619ee96b6" containerName="revoke-keys" Nov 03 16:08:48.451167 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.451150 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" Nov 03 16:08:48.511252 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.511210 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:08:48.511442 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.511264 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 16:08:48.514962 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.514937 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:08:48.543686 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.543645 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e52477-93a8-41f6-a49a-6a04a5650636-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64\" (UID: \"67e52477-93a8-41f6-a49a-6a04a5650636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" Nov 03 16:08:48.543884 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.543773 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhxjs\" (UniqueName: \"kubernetes.io/projected/67e52477-93a8-41f6-a49a-6a04a5650636-kube-api-access-hhxjs\") pod \"maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64\" (UID: \"67e52477-93a8-41f6-a49a-6a04a5650636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" Nov 03 16:08:48.543884 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.543808 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e52477-93a8-41f6-a49a-6a04a5650636-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64\" (UID: \"67e52477-93a8-41f6-a49a-6a04a5650636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" Nov 03 16:08:48.561265 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.561228 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:08:48.564538 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.564514 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64"] Nov 03 16:08:48.644664 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.644623 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e52477-93a8-41f6-a49a-6a04a5650636-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64\" (UID: \"67e52477-93a8-41f6-a49a-6a04a5650636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" Nov 03 16:08:48.644909 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.644683 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxjs\" (UniqueName: \"kubernetes.io/projected/67e52477-93a8-41f6-a49a-6a04a5650636-kube-api-access-hhxjs\") pod \"maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64\" (UID: \"67e52477-93a8-41f6-a49a-6a04a5650636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" Nov 03 16:08:48.644909 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.644745 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e52477-93a8-41f6-a49a-6a04a5650636-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64\" (UID: \"67e52477-93a8-41f6-a49a-6a04a5650636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" Nov 03 16:08:48.645416 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.645395 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e52477-93a8-41f6-a49a-6a04a5650636-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64\" (UID: \"67e52477-93a8-41f6-a49a-6a04a5650636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" Nov 03 16:08:48.647243 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.647220 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e52477-93a8-41f6-a49a-6a04a5650636-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64\" (UID: \"67e52477-93a8-41f6-a49a-6a04a5650636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" Nov 03 16:08:48.707099 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.707019 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhxjs\" (UniqueName: \"kubernetes.io/projected/67e52477-93a8-41f6-a49a-6a04a5650636-kube-api-access-hhxjs\") pod \"maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64\" (UID: \"67e52477-93a8-41f6-a49a-6a04a5650636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" Nov 03 16:08:48.765385 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:48.765352 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" Nov 03 16:08:49.145110 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:49.145081 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64"] Nov 03 16:08:49.146263 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:08:49.146237 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e52477_93a8_41f6_a49a_6a04a5650636.slice/crio-d3c1734444910b6014ceaa33d9a5643805ef6003960346898e011c2359c14e41 WatchSource:0}: Error finding container d3c1734444910b6014ceaa33d9a5643805ef6003960346898e011c2359c14e41: Status 404 returned error can't find the container with id d3c1734444910b6014ceaa33d9a5643805ef6003960346898e011c2359c14e41 Nov 03 16:08:49.536977 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:49.536942 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" event={"ID":"67e52477-93a8-41f6-a49a-6a04a5650636","Type":"ContainerStarted","Data":"b60ec90ab80fcec35d5c2c4ea65262af66b9ee2042c10d2307e2f8dd2dbc8cf2"} Nov 03 16:08:49.537479 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:49.537003 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" event={"ID":"67e52477-93a8-41f6-a49a-6a04a5650636","Type":"ContainerStarted","Data":"d3c1734444910b6014ceaa33d9a5643805ef6003960346898e011c2359c14e41"} Nov 03 16:08:49.656453 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:49.656395 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" podStartSLOduration=1.6563745220000001 podStartE2EDuration="1.656374522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:08:49.654535999 +0000 UTC m=+3637.085493805" watchObservedRunningTime="2026-11-03 16:08:49.656374522 +0000 UTC m=+3637.087332328" Nov 03 16:08:50.545987 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:50.545952 4290 generic.go:356] "Generic (PLEG): container finished" podID="67e52477-93a8-41f6-a49a-6a04a5650636" containerID="b60ec90ab80fcec35d5c2c4ea65262af66b9ee2042c10d2307e2f8dd2dbc8cf2" exitCode=0 Nov 03 16:08:50.546393 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:50.546032 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" event={"ID":"67e52477-93a8-41f6-a49a-6a04a5650636","Type":"ContainerDied","Data":"b60ec90ab80fcec35d5c2c4ea65262af66b9ee2042c10d2307e2f8dd2dbc8cf2"} Nov 03 16:08:51.691220 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:51.691196 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" Nov 03 16:08:51.777496 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:51.777463 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67e52477-93a8-41f6-a49a-6a04a5650636-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e52477-93a8-41f6-a49a-6a04a5650636-maas-api-xks-ca\") pod \"67e52477-93a8-41f6-a49a-6a04a5650636\" (UID: \"67e52477-93a8-41f6-a49a-6a04a5650636\") " Nov 03 16:08:51.777677 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:51.777521 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e52477-93a8-41f6-a49a-6a04a5650636-kube-api-access-hhxjs\" (UniqueName: \"kubernetes.io/projected/67e52477-93a8-41f6-a49a-6a04a5650636-kube-api-access-hhxjs\") pod \"67e52477-93a8-41f6-a49a-6a04a5650636\" (UID: \"67e52477-93a8-41f6-a49a-6a04a5650636\") " Nov 03 16:08:51.777677 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:51.777586 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67e52477-93a8-41f6-a49a-6a04a5650636-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e52477-93a8-41f6-a49a-6a04a5650636-maas-api-openshift-ca\") pod \"67e52477-93a8-41f6-a49a-6a04a5650636\" (UID: \"67e52477-93a8-41f6-a49a-6a04a5650636\") " Nov 03 16:08:51.778000 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:51.777963 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e52477-93a8-41f6-a49a-6a04a5650636-maas-api-openshift-ca" pod "67e52477-93a8-41f6-a49a-6a04a5650636" (UID: "67e52477-93a8-41f6-a49a-6a04a5650636"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:08:51.779849 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:51.779822 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e52477-93a8-41f6-a49a-6a04a5650636-maas-api-xks-ca" pod "67e52477-93a8-41f6-a49a-6a04a5650636" (UID: "67e52477-93a8-41f6-a49a-6a04a5650636"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:08:51.779992 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:51.779967 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e52477-93a8-41f6-a49a-6a04a5650636-kube-api-access-hhxjs" pod "67e52477-93a8-41f6-a49a-6a04a5650636" (UID: "67e52477-93a8-41f6-a49a-6a04a5650636"). InnerVolumeSpecName "kube-api-access-hhxjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:08:51.878675 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:51.878644 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e52477-93a8-41f6-a49a-6a04a5650636-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:08:51.878675 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:51.878675 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e52477-93a8-41f6-a49a-6a04a5650636-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:08:51.878902 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:51.878685 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhxjs\" (UniqueName: \"kubernetes.io/projected/67e52477-93a8-41f6-a49a-6a04a5650636-kube-api-access-hhxjs\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:08:52.562008 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:52.561967 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" event={"ID":"67e52477-93a8-41f6-a49a-6a04a5650636","Type":"ContainerDied","Data":"d3c1734444910b6014ceaa33d9a5643805ef6003960346898e011c2359c14e41"} Nov 03 16:08:52.562178 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:52.562047 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3c1734444910b6014ceaa33d9a5643805ef6003960346898e011c2359c14e41" Nov 03 16:08:52.562178 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:08:52.561984 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64" Nov 03 16:09:08.702480 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:09:08.702386 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:09:15.666676 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:15.666645 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64"] Nov 03 16:09:15.716411 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:15.716375 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-877924-q6f64"] Nov 03 16:09:16.677451 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:16.677417 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e52477-93a8-41f6-a49a-6a04a5650636" path="/var/lib/kubelet/pods/67e52477-93a8-41f6-a49a-6a04a5650636/volumes" Nov 03 16:09:18.770025 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:18.769992 4290 scope.go:122] "RemoveContainer" containerID="be226af41e41c4d44d7b6f4d60c8b5599ae6ce7adb3dd90f2d8f503edb46eced" Nov 03 16:09:18.780750 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:18.780730 4290 scope.go:122] "RemoveContainer" containerID="2ca9f413ab8f2a7495bf2e38d115e336f5f6bfa435ec8e3a2b15a2738d74a250" Nov 03 16:09:38.703436 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:09:38.703398 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:09:49.054126 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:49.054089 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k"] Nov 03 16:09:49.103628 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:49.103595 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7p25k"] Nov 03 16:09:50.562616 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.562572 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt"] Nov 03 16:09:50.563319 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.563297 4290 cpu_manager.go:414] "containerMap: removing container" podUID="67e52477-93a8-41f6-a49a-6a04a5650636" containerName="revoke-keys" Nov 03 16:09:50.563420 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.563320 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e52477-93a8-41f6-a49a-6a04a5650636" containerName="revoke-keys" Nov 03 16:09:50.563501 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.563487 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e52477-93a8-41f6-a49a-6a04a5650636" containerName="revoke-keys" Nov 03 16:09:50.566375 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.566353 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" Nov 03 16:09:50.625180 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.625147 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt"] Nov 03 16:09:50.632324 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.632287 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn692\" (UniqueName: \"kubernetes.io/projected/7af44f6b-a93b-40a6-bc84-a88045776354-kube-api-access-hn692\") pod \"maas-api-revoke-sub-f03629db-wfwnt\" (UID: \"7af44f6b-a93b-40a6-bc84-a88045776354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" Nov 03 16:09:50.632324 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.632326 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af44f6b-a93b-40a6-bc84-a88045776354-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wfwnt\" (UID: \"7af44f6b-a93b-40a6-bc84-a88045776354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" Nov 03 16:09:50.632585 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.632479 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af44f6b-a93b-40a6-bc84-a88045776354-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wfwnt\" (UID: \"7af44f6b-a93b-40a6-bc84-a88045776354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" Nov 03 16:09:50.671187 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.671161 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:09:50.671578 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.671558 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:09:50.671718 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.671663 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:09:50.671787 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.671773 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 16:09:50.677633 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.677611 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2055023e-998a-4aba-a4b5-942fcec069da" path="/var/lib/kubelet/pods/2055023e-998a-4aba-a4b5-942fcec069da/volumes" Nov 03 16:09:50.733370 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.733342 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn692\" (UniqueName: \"kubernetes.io/projected/7af44f6b-a93b-40a6-bc84-a88045776354-kube-api-access-hn692\") pod \"maas-api-revoke-sub-f03629db-wfwnt\" (UID: \"7af44f6b-a93b-40a6-bc84-a88045776354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" Nov 03 16:09:50.733650 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.733588 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af44f6b-a93b-40a6-bc84-a88045776354-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wfwnt\" (UID: \"7af44f6b-a93b-40a6-bc84-a88045776354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" Nov 03 16:09:50.733883 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.733846 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af44f6b-a93b-40a6-bc84-a88045776354-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wfwnt\" (UID: \"7af44f6b-a93b-40a6-bc84-a88045776354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" Nov 03 16:09:50.734524 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.734503 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af44f6b-a93b-40a6-bc84-a88045776354-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wfwnt\" (UID: \"7af44f6b-a93b-40a6-bc84-a88045776354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" Nov 03 16:09:50.736678 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.736653 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af44f6b-a93b-40a6-bc84-a88045776354-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wfwnt\" (UID: \"7af44f6b-a93b-40a6-bc84-a88045776354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" Nov 03 16:09:50.796140 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.796112 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn692\" (UniqueName: \"kubernetes.io/projected/7af44f6b-a93b-40a6-bc84-a88045776354-kube-api-access-hn692\") pod \"maas-api-revoke-sub-f03629db-wfwnt\" (UID: \"7af44f6b-a93b-40a6-bc84-a88045776354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" Nov 03 16:09:50.879827 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:50.879806 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" Nov 03 16:09:51.258269 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:51.258239 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt"] Nov 03 16:09:51.259538 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:09:51.259510 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af44f6b_a93b_40a6_bc84_a88045776354.slice/crio-ab9069651d038477c656e7f80b4bc593c94bf033fbcde63ced38c358a512c924 WatchSource:0}: Error finding container ab9069651d038477c656e7f80b4bc593c94bf033fbcde63ced38c358a512c924: Status 404 returned error can't find the container with id ab9069651d038477c656e7f80b4bc593c94bf033fbcde63ced38c358a512c924 Nov 03 16:09:51.979563 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:51.979528 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" event={"ID":"7af44f6b-a93b-40a6-bc84-a88045776354","Type":"ContainerStarted","Data":"0bed0c528d0bacbb3e1e88d3743ebec99c09b90886487700f121ea64ace6f0c2"} Nov 03 16:09:51.979996 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:51.979580 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" event={"ID":"7af44f6b-a93b-40a6-bc84-a88045776354","Type":"ContainerStarted","Data":"ab9069651d038477c656e7f80b4bc593c94bf033fbcde63ced38c358a512c924"} Nov 03 16:09:52.098014 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:52.097959 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" podStartSLOduration=2.097945943 podStartE2EDuration="2.097945943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:09:52.094950426 +0000 UTC m=+3699.525908232" watchObservedRunningTime="2026-11-03 16:09:52.097945943 +0000 UTC m=+3699.528903748" Nov 03 16:09:52.988533 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:52.988501 4290 generic.go:356] "Generic (PLEG): container finished" podID="7af44f6b-a93b-40a6-bc84-a88045776354" containerID="0bed0c528d0bacbb3e1e88d3743ebec99c09b90886487700f121ea64ace6f0c2" exitCode=0 Nov 03 16:09:52.988994 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:52.988594 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" event={"ID":"7af44f6b-a93b-40a6-bc84-a88045776354","Type":"ContainerDied","Data":"0bed0c528d0bacbb3e1e88d3743ebec99c09b90886487700f121ea64ace6f0c2"} Nov 03 16:09:54.156336 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:54.156305 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" Nov 03 16:09:54.269903 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:54.269822 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af44f6b-a93b-40a6-bc84-a88045776354-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af44f6b-a93b-40a6-bc84-a88045776354-maas-api-openshift-ca\") pod \"7af44f6b-a93b-40a6-bc84-a88045776354\" (UID: \"7af44f6b-a93b-40a6-bc84-a88045776354\") " Nov 03 16:09:54.269903 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:54.269892 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af44f6b-a93b-40a6-bc84-a88045776354-kube-api-access-hn692\" (UniqueName: \"kubernetes.io/projected/7af44f6b-a93b-40a6-bc84-a88045776354-kube-api-access-hn692\") pod \"7af44f6b-a93b-40a6-bc84-a88045776354\" (UID: \"7af44f6b-a93b-40a6-bc84-a88045776354\") " Nov 03 16:09:54.270100 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:54.269947 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7af44f6b-a93b-40a6-bc84-a88045776354-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af44f6b-a93b-40a6-bc84-a88045776354-maas-api-xks-ca\") pod \"7af44f6b-a93b-40a6-bc84-a88045776354\" (UID: \"7af44f6b-a93b-40a6-bc84-a88045776354\") " Nov 03 16:09:54.270211 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:54.270190 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af44f6b-a93b-40a6-bc84-a88045776354-maas-api-openshift-ca" pod "7af44f6b-a93b-40a6-bc84-a88045776354" (UID: "7af44f6b-a93b-40a6-bc84-a88045776354"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:09:54.270367 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:54.270348 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af44f6b-a93b-40a6-bc84-a88045776354-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:09:54.272157 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:54.272127 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af44f6b-a93b-40a6-bc84-a88045776354-maas-api-xks-ca" pod "7af44f6b-a93b-40a6-bc84-a88045776354" (UID: "7af44f6b-a93b-40a6-bc84-a88045776354"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:09:54.272270 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:54.272203 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af44f6b-a93b-40a6-bc84-a88045776354-kube-api-access-hn692" pod "7af44f6b-a93b-40a6-bc84-a88045776354" (UID: "7af44f6b-a93b-40a6-bc84-a88045776354"). InnerVolumeSpecName "kube-api-access-hn692". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:09:54.371381 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:54.371347 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn692\" (UniqueName: \"kubernetes.io/projected/7af44f6b-a93b-40a6-bc84-a88045776354-kube-api-access-hn692\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:09:54.371381 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:54.371380 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af44f6b-a93b-40a6-bc84-a88045776354-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:09:55.002841 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:55.002748 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" event={"ID":"7af44f6b-a93b-40a6-bc84-a88045776354","Type":"ContainerDied","Data":"ab9069651d038477c656e7f80b4bc593c94bf033fbcde63ced38c358a512c924"} Nov 03 16:09:55.002841 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:55.002777 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wfwnt" Nov 03 16:09:55.002841 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:09:55.002807 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab9069651d038477c656e7f80b4bc593c94bf033fbcde63ced38c358a512c924" Nov 03 16:10:08.710405 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:10:08.710307 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:10:10.778725 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:10.777912 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-wl2z4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:10:10.778725 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:10.777998 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wl2z4" podUID="a3cec29f-05df-404d-b346-53dda11adf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:10:11.824251 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:11.824219 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk"] Nov 03 16:10:11.824741 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:11.824725 4290 cpu_manager.go:414] "containerMap: removing container" podUID="7af44f6b-a93b-40a6-bc84-a88045776354" containerName="revoke-keys" Nov 03 16:10:11.824792 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:11.824741 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af44f6b-a93b-40a6-bc84-a88045776354" containerName="revoke-keys" Nov 03 16:10:11.824829 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:11.824814 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af44f6b-a93b-40a6-bc84-a88045776354" containerName="revoke-keys" Nov 03 16:10:11.828051 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:11.828034 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" Nov 03 16:10:11.883166 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:11.883125 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:10:11.922953 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:11.922926 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f244b591-7df7-4d26-88ec-946bae271536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-ctsgk\" (UID: \"f244b591-7df7-4d26-88ec-946bae271536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" Nov 03 16:10:11.923133 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:11.922988 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfr8x\" (UniqueName: \"kubernetes.io/projected/f244b591-7df7-4d26-88ec-946bae271536-kube-api-access-tfr8x\") pod \"maas-api-revoke-sub-0165d7d2-ctsgk\" (UID: \"f244b591-7df7-4d26-88ec-946bae271536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" Nov 03 16:10:11.923133 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:11.923055 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f244b591-7df7-4d26-88ec-946bae271536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-ctsgk\" (UID: \"f244b591-7df7-4d26-88ec-946bae271536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" Nov 03 16:10:11.932765 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:11.932740 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:10:11.932921 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:11.932874 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:10:11.932921 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:11.932911 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 16:10:11.936125 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:11.936103 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk"] Nov 03 16:10:12.024189 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:12.024159 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f244b591-7df7-4d26-88ec-946bae271536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-ctsgk\" (UID: \"f244b591-7df7-4d26-88ec-946bae271536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" Nov 03 16:10:12.024348 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:12.024225 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfr8x\" (UniqueName: \"kubernetes.io/projected/f244b591-7df7-4d26-88ec-946bae271536-kube-api-access-tfr8x\") pod \"maas-api-revoke-sub-0165d7d2-ctsgk\" (UID: \"f244b591-7df7-4d26-88ec-946bae271536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" Nov 03 16:10:12.024348 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:12.024243 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f244b591-7df7-4d26-88ec-946bae271536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-ctsgk\" (UID: \"f244b591-7df7-4d26-88ec-946bae271536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" Nov 03 16:10:12.024998 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:12.024971 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f244b591-7df7-4d26-88ec-946bae271536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-ctsgk\" (UID: \"f244b591-7df7-4d26-88ec-946bae271536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" Nov 03 16:10:12.026893 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:12.026872 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f244b591-7df7-4d26-88ec-946bae271536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-ctsgk\" (UID: \"f244b591-7df7-4d26-88ec-946bae271536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" Nov 03 16:10:12.085864 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:12.085798 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfr8x\" (UniqueName: \"kubernetes.io/projected/f244b591-7df7-4d26-88ec-946bae271536-kube-api-access-tfr8x\") pod \"maas-api-revoke-sub-0165d7d2-ctsgk\" (UID: \"f244b591-7df7-4d26-88ec-946bae271536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" Nov 03 16:10:12.141514 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:12.141491 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" Nov 03 16:10:12.536453 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:12.536424 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk"] Nov 03 16:10:12.537271 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:10:12.537240 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf244b591_7df7_4d26_88ec_946bae271536.slice/crio-fcfa181efdb6bbc8fce40433e633b313185cea0115f445018e3af95007ec9fbf WatchSource:0}: Error finding container fcfa181efdb6bbc8fce40433e633b313185cea0115f445018e3af95007ec9fbf: Status 404 returned error can't find the container with id fcfa181efdb6bbc8fce40433e633b313185cea0115f445018e3af95007ec9fbf Nov 03 16:10:13.138232 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:13.138193 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" event={"ID":"f244b591-7df7-4d26-88ec-946bae271536","Type":"ContainerStarted","Data":"0dae79c6f31626c780647c0ebaf70069b7d0c32e3d01f46272060c0e955b4686"} Nov 03 16:10:13.138645 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:13.138260 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" event={"ID":"f244b591-7df7-4d26-88ec-946bae271536","Type":"ContainerStarted","Data":"fcfa181efdb6bbc8fce40433e633b313185cea0115f445018e3af95007ec9fbf"} Nov 03 16:10:13.259219 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:13.259159 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" podStartSLOduration=2.259144275 podStartE2EDuration="2.259144275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:13.255684397 +0000 UTC m=+3720.686642204" watchObservedRunningTime="2026-11-03 16:10:13.259144275 +0000 UTC m=+3720.690102081" Nov 03 16:10:14.147071 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:14.147038 4290 generic.go:356] "Generic (PLEG): container finished" podID="f244b591-7df7-4d26-88ec-946bae271536" containerID="0dae79c6f31626c780647c0ebaf70069b7d0c32e3d01f46272060c0e955b4686" exitCode=0 Nov 03 16:10:14.147497 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:14.147130 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" event={"ID":"f244b591-7df7-4d26-88ec-946bae271536","Type":"ContainerDied","Data":"0dae79c6f31626c780647c0ebaf70069b7d0c32e3d01f46272060c0e955b4686"} Nov 03 16:10:15.296987 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:15.296962 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" Nov 03 16:10:15.455083 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:15.455003 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f244b591-7df7-4d26-88ec-946bae271536-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f244b591-7df7-4d26-88ec-946bae271536-maas-api-openshift-ca\") pod \"f244b591-7df7-4d26-88ec-946bae271536\" (UID: \"f244b591-7df7-4d26-88ec-946bae271536\") " Nov 03 16:10:15.455233 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:15.455189 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f244b591-7df7-4d26-88ec-946bae271536-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f244b591-7df7-4d26-88ec-946bae271536-maas-api-xks-ca\") pod \"f244b591-7df7-4d26-88ec-946bae271536\" (UID: \"f244b591-7df7-4d26-88ec-946bae271536\") " Nov 03 16:10:15.455233 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:15.455227 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f244b591-7df7-4d26-88ec-946bae271536-kube-api-access-tfr8x\" (UniqueName: \"kubernetes.io/projected/f244b591-7df7-4d26-88ec-946bae271536-kube-api-access-tfr8x\") pod \"f244b591-7df7-4d26-88ec-946bae271536\" (UID: \"f244b591-7df7-4d26-88ec-946bae271536\") " Nov 03 16:10:15.455381 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:15.455355 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f244b591-7df7-4d26-88ec-946bae271536-maas-api-openshift-ca" pod "f244b591-7df7-4d26-88ec-946bae271536" (UID: "f244b591-7df7-4d26-88ec-946bae271536"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:10:15.455577 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:15.455559 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f244b591-7df7-4d26-88ec-946bae271536-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:10:15.457359 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:15.457335 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f244b591-7df7-4d26-88ec-946bae271536-maas-api-xks-ca" pod "f244b591-7df7-4d26-88ec-946bae271536" (UID: "f244b591-7df7-4d26-88ec-946bae271536"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:10:15.457488 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:15.457471 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f244b591-7df7-4d26-88ec-946bae271536-kube-api-access-tfr8x" pod "f244b591-7df7-4d26-88ec-946bae271536" (UID: "f244b591-7df7-4d26-88ec-946bae271536"). InnerVolumeSpecName "kube-api-access-tfr8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:10:15.556234 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:15.556200 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f244b591-7df7-4d26-88ec-946bae271536-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:10:15.556234 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:15.556228 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfr8x\" (UniqueName: \"kubernetes.io/projected/f244b591-7df7-4d26-88ec-946bae271536-kube-api-access-tfr8x\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:10:16.163469 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:16.163441 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" Nov 03 16:10:16.163675 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:16.163470 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ctsgk" event={"ID":"f244b591-7df7-4d26-88ec-946bae271536","Type":"ContainerDied","Data":"fcfa181efdb6bbc8fce40433e633b313185cea0115f445018e3af95007ec9fbf"} Nov 03 16:10:16.163675 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:16.163534 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcfa181efdb6bbc8fce40433e633b313185cea0115f445018e3af95007ec9fbf" Nov 03 16:10:23.655779 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.655732 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j"] Nov 03 16:10:23.656267 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.656225 4290 cpu_manager.go:414] "containerMap: removing container" podUID="f244b591-7df7-4d26-88ec-946bae271536" containerName="revoke-keys" Nov 03 16:10:23.656267 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.656237 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f244b591-7df7-4d26-88ec-946bae271536" containerName="revoke-keys" Nov 03 16:10:23.656346 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.656324 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="f244b591-7df7-4d26-88ec-946bae271536" containerName="revoke-keys" Nov 03 16:10:23.659664 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.659648 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" Nov 03 16:10:23.740712 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.740661 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d738eb-d32e-456c-b920-7b03b49d0af9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xzf2j\" (UID: \"b0d738eb-d32e-456c-b920-7b03b49d0af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" Nov 03 16:10:23.740882 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.740723 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf8dx\" (UniqueName: \"kubernetes.io/projected/b0d738eb-d32e-456c-b920-7b03b49d0af9-kube-api-access-bf8dx\") pod \"maas-api-revoke-sub-6b19338f-xzf2j\" (UID: \"b0d738eb-d32e-456c-b920-7b03b49d0af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" Nov 03 16:10:23.740882 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.740850 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d738eb-d32e-456c-b920-7b03b49d0af9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xzf2j\" (UID: \"b0d738eb-d32e-456c-b920-7b03b49d0af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" Nov 03 16:10:23.763619 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.763595 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:10:23.763824 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.763691 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:10:23.763824 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.763739 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 16:10:23.763824 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.763728 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:10:23.766955 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.766936 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j"] Nov 03 16:10:23.842430 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.842400 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d738eb-d32e-456c-b920-7b03b49d0af9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xzf2j\" (UID: \"b0d738eb-d32e-456c-b920-7b03b49d0af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" Nov 03 16:10:23.842624 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.842438 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf8dx\" (UniqueName: \"kubernetes.io/projected/b0d738eb-d32e-456c-b920-7b03b49d0af9-kube-api-access-bf8dx\") pod \"maas-api-revoke-sub-6b19338f-xzf2j\" (UID: \"b0d738eb-d32e-456c-b920-7b03b49d0af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" Nov 03 16:10:23.842624 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.842473 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d738eb-d32e-456c-b920-7b03b49d0af9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xzf2j\" (UID: \"b0d738eb-d32e-456c-b920-7b03b49d0af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" Nov 03 16:10:23.843441 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.843380 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d738eb-d32e-456c-b920-7b03b49d0af9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xzf2j\" (UID: \"b0d738eb-d32e-456c-b920-7b03b49d0af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" Nov 03 16:10:23.845108 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.845080 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d738eb-d32e-456c-b920-7b03b49d0af9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xzf2j\" (UID: \"b0d738eb-d32e-456c-b920-7b03b49d0af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" Nov 03 16:10:23.904049 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.904009 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf8dx\" (UniqueName: \"kubernetes.io/projected/b0d738eb-d32e-456c-b920-7b03b49d0af9-kube-api-access-bf8dx\") pod \"maas-api-revoke-sub-6b19338f-xzf2j\" (UID: \"b0d738eb-d32e-456c-b920-7b03b49d0af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" Nov 03 16:10:23.974088 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:23.974015 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" Nov 03 16:10:24.362914 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:24.362886 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j"] Nov 03 16:10:24.365089 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:10:24.365055 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d738eb_d32e_456c_b920_7b03b49d0af9.slice/crio-61c69ef16f53d0c6a252d8ae27934b1df9dda7a5acad4c06733f702f76a93fc4 WatchSource:0}: Error finding container 61c69ef16f53d0c6a252d8ae27934b1df9dda7a5acad4c06733f702f76a93fc4: Status 404 returned error can't find the container with id 61c69ef16f53d0c6a252d8ae27934b1df9dda7a5acad4c06733f702f76a93fc4 Nov 03 16:10:25.223281 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:25.223256 4290 generic.go:356] "Generic (PLEG): container finished" podID="b0d738eb-d32e-456c-b920-7b03b49d0af9" containerID="cdf40dba05a1fc806e521a426b8cf06c10da15ef812ac87d7df1761fad381e32" exitCode=0 Nov 03 16:10:25.223612 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:25.223337 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" event={"ID":"b0d738eb-d32e-456c-b920-7b03b49d0af9","Type":"ContainerDied","Data":"cdf40dba05a1fc806e521a426b8cf06c10da15ef812ac87d7df1761fad381e32"} Nov 03 16:10:25.223612 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:25.223391 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" event={"ID":"b0d738eb-d32e-456c-b920-7b03b49d0af9","Type":"ContainerStarted","Data":"61c69ef16f53d0c6a252d8ae27934b1df9dda7a5acad4c06733f702f76a93fc4"} Nov 03 16:10:26.370644 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:26.370620 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" Nov 03 16:10:26.468892 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:26.468857 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0d738eb-d32e-456c-b920-7b03b49d0af9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d738eb-d32e-456c-b920-7b03b49d0af9-maas-api-xks-ca\") pod \"b0d738eb-d32e-456c-b920-7b03b49d0af9\" (UID: \"b0d738eb-d32e-456c-b920-7b03b49d0af9\") " Nov 03 16:10:26.468892 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:26.468894 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0d738eb-d32e-456c-b920-7b03b49d0af9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d738eb-d32e-456c-b920-7b03b49d0af9-maas-api-openshift-ca\") pod \"b0d738eb-d32e-456c-b920-7b03b49d0af9\" (UID: \"b0d738eb-d32e-456c-b920-7b03b49d0af9\") " Nov 03 16:10:26.469181 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:26.469018 4290 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d738eb-d32e-456c-b920-7b03b49d0af9-kube-api-access-bf8dx\" (UniqueName: \"kubernetes.io/projected/b0d738eb-d32e-456c-b920-7b03b49d0af9-kube-api-access-bf8dx\") pod \"b0d738eb-d32e-456c-b920-7b03b49d0af9\" (UID: \"b0d738eb-d32e-456c-b920-7b03b49d0af9\") " Nov 03 16:10:26.469331 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:26.469303 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d738eb-d32e-456c-b920-7b03b49d0af9-maas-api-openshift-ca" pod "b0d738eb-d32e-456c-b920-7b03b49d0af9" (UID: "b0d738eb-d32e-456c-b920-7b03b49d0af9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:10:26.471151 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:26.471129 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d738eb-d32e-456c-b920-7b03b49d0af9-maas-api-xks-ca" pod "b0d738eb-d32e-456c-b920-7b03b49d0af9" (UID: "b0d738eb-d32e-456c-b920-7b03b49d0af9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:10:26.471236 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:26.471226 4290 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d738eb-d32e-456c-b920-7b03b49d0af9-kube-api-access-bf8dx" pod "b0d738eb-d32e-456c-b920-7b03b49d0af9" (UID: "b0d738eb-d32e-456c-b920-7b03b49d0af9"). InnerVolumeSpecName "kube-api-access-bf8dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:10:26.570308 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:26.570197 4290 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf8dx\" (UniqueName: \"kubernetes.io/projected/b0d738eb-d32e-456c-b920-7b03b49d0af9-kube-api-access-bf8dx\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:10:26.570308 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:26.570249 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d738eb-d32e-456c-b920-7b03b49d0af9-maas-api-xks-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:10:26.570308 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:26.570259 4290 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d738eb-d32e-456c-b920-7b03b49d0af9-maas-api-openshift-ca\") on node \"ip-10-0-139-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:10:27.237590 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:27.237551 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" event={"ID":"b0d738eb-d32e-456c-b920-7b03b49d0af9","Type":"ContainerDied","Data":"61c69ef16f53d0c6a252d8ae27934b1df9dda7a5acad4c06733f702f76a93fc4"} Nov 03 16:10:27.237590 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:27.237575 4290 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xzf2j" Nov 03 16:10:27.237925 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:27.237632 4290 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61c69ef16f53d0c6a252d8ae27934b1df9dda7a5acad4c06733f702f76a93fc4" Nov 03 16:10:35.906553 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:35.906522 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b7867b55-9l2fl_8c0f5828-4470-45ae-af7b-3e2defa2bb3e/authorino/0.log" Nov 03 16:10:37.997934 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:37.997894 4290 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-2sbg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:10:37.998589 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:37.998554 4290 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-2sbg2" podUID="871baa2b-806e-4033-a553-277c509a3c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:10:38.701165 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:10:38.701127 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:10:47.650816 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:47.650782 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-59f8d677c6-hwwtv_060263ec-48f6-48ab-b4aa-04aa578327ec/manager/0.log" Nov 03 16:10:48.334946 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:48.334912 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-24ckp_f41479d0-68c6-43c3-8a51-bfef6bcfe828/manager/0.log" Nov 03 16:10:49.051019 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:49.050981 4290 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x"] Nov 03 16:10:49.100616 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:49.100580 4290 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a8d6c0-5t24x"] Nov 03 16:10:49.177023 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:49.176991 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65f6786cb5-5wwgn_83bb0c49-dd3e-4323-b755-b78034bd8c30/manager/0.log" Nov 03 16:10:50.567676 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:50.567639 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hcsvr_52901e30-cdd3-42e9-9063-20dc5b2f2c66/proxy/0.log" Nov 03 16:10:50.677498 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:50.677468 4290 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd" path="/var/lib/kubelet/pods/dc1f5e6c-cf01-4c7e-9b48-73a937fd18dd/volumes" Nov 03 16:10:52.160424 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:52.160390 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-ctsgk_f244b591-7df7-4d26-88ec-946bae271536/revoke-keys/0.log" Nov 03 16:10:52.416692 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:52.416612 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-xzf2j_b0d738eb-d32e-456c-b920-7b03b49d0af9/revoke-keys/0.log" Nov 03 16:10:52.657353 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:52.657321 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-wfwnt_7af44f6b-a93b-40a6-bc84-a88045776354/revoke-keys/0.log" Nov 03 16:10:54.574924 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:54.574890 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b7867b55-9l2fl_8c0f5828-4470-45ae-af7b-3e2defa2bb3e/authorino/0.log" Nov 03 16:10:55.053025 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:55.052993 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8tnvm_36653232-724e-4a78-b661-5a5c6adc541e/manager/0.log" Nov 03 16:10:55.881254 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:55.881223 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-fzcg4_cdfe745b-0aad-43f7-aa54-1c44d45bde19/manager/0.log" Nov 03 16:10:57.705216 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:57.705182 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6644c85865t8j8h_f9ee4847-f5a9-4549-ba96-3e7a9b79db99/istio-proxy/0.log" Nov 03 16:10:58.935777 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:10:58.935741 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-2sbg2_871baa2b-806e-4033-a553-277c509a3c14/istio-proxy/1.log" Nov 03 16:11:00.870668 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:00.870638 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-pvhdp_c56971b8-db79-464a-bdaa-1a1e950f7c80/istio-proxy/0.log" Nov 03 16:11:01.812865 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:01.812836 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-wl2z4_a3cec29f-05df-404d-b346-53dda11adf8c/istio-proxy/0.log" Nov 03 16:11:02.162255 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:02.162224 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z_b5daa2f8-b704-445c-b085-ad6fb0b52063/istio-proxy/0.log" Nov 03 16:11:02.393289 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:02.393261 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-vps22_d37b293a-544b-4ca5-9810-8bdbc888ca29/payload-pre-processing/0.log" Nov 03 16:11:02.636142 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:02.636114 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-f7dkj_0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8/payload-processing/0.log" Nov 03 16:11:03.498475 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:03.498438 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-2sbg2_871baa2b-806e-4033-a553-277c509a3c14/istio-proxy/0.log" Nov 03 16:11:04.457231 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:04.457196 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7qzlq_208146e8-fd8f-4d34-8e96-61c16cd5d4bd/main/0.log" Nov 03 16:11:05.271433 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:05.271402 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td_3f31f078-0575-4510-bee2-628a20f0caad/main/0.log" Nov 03 16:11:05.558680 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:05.558588 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qqz7l_72f19618-4cf9-49cf-8416-187d0de0cf0b/main/0.log" Nov 03 16:11:07.195907 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:07.195874 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b7867b55-9l2fl_8c0f5828-4470-45ae-af7b-3e2defa2bb3e/authorino/0.log" Nov 03 16:11:08.703107 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:11:08.703019 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:11:17.233098 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:17.233065 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-59f8d677c6-hwwtv_060263ec-48f6-48ab-b4aa-04aa578327ec/manager/0.log" Nov 03 16:11:17.920479 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:17.920449 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-24ckp_f41479d0-68c6-43c3-8a51-bfef6bcfe828/manager/0.log" Nov 03 16:11:18.662910 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:18.662876 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65f6786cb5-5wwgn_83bb0c49-dd3e-4323-b755-b78034bd8c30/manager/0.log" Nov 03 16:11:18.818768 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:18.818737 4290 scope.go:122] "RemoveContainer" containerID="6f27a129cbd9dbb18d11c3e0c158ade74dd068b146760f9282296fc0216b3192" Nov 03 16:11:19.655922 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:19.655853 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hcsvr_52901e30-cdd3-42e9-9063-20dc5b2f2c66/proxy/0.log" Nov 03 16:11:20.942248 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:20.942219 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-ctsgk_f244b591-7df7-4d26-88ec-946bae271536/revoke-keys/0.log" Nov 03 16:11:21.177638 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:21.177608 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-xzf2j_b0d738eb-d32e-456c-b920-7b03b49d0af9/revoke-keys/0.log" Nov 03 16:11:21.407512 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:21.407483 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-wfwnt_7af44f6b-a93b-40a6-bc84-a88045776354/revoke-keys/0.log" Nov 03 16:11:22.908407 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:22.908365 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b7867b55-9l2fl_8c0f5828-4470-45ae-af7b-3e2defa2bb3e/authorino/0.log" Nov 03 16:11:23.396487 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:23.396453 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8tnvm_36653232-724e-4a78-b661-5a5c6adc541e/manager/0.log" Nov 03 16:11:24.181541 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:24.181502 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-fzcg4_cdfe745b-0aad-43f7-aa54-1c44d45bde19/manager/0.log" Nov 03 16:11:25.502985 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:25.502952 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6644c85865t8j8h_f9ee4847-f5a9-4549-ba96-3e7a9b79db99/istio-proxy/0.log" Nov 03 16:11:26.626098 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:26.626067 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-2sbg2_871baa2b-806e-4033-a553-277c509a3c14/istio-proxy/1.log" Nov 03 16:11:28.397860 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:28.397814 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-pvhdp_c56971b8-db79-464a-bdaa-1a1e950f7c80/istio-proxy/0.log" Nov 03 16:11:29.319898 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:29.319872 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-wl2z4_a3cec29f-05df-404d-b346-53dda11adf8c/istio-proxy/0.log" Nov 03 16:11:29.651711 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:29.651663 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cb6869599-ckq2z_b5daa2f8-b704-445c-b085-ad6fb0b52063/istio-proxy/0.log" Nov 03 16:11:29.874853 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:29.874820 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-vps22_d37b293a-544b-4ca5-9810-8bdbc888ca29/payload-pre-processing/0.log" Nov 03 16:11:30.111132 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:30.111042 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-f7dkj_0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8/payload-processing/0.log" Nov 03 16:11:31.327840 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:31.327806 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7qzlq_208146e8-fd8f-4d34-8e96-61c16cd5d4bd/main/0.log" Nov 03 16:11:32.157475 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:32.157442 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k95td_3f31f078-0575-4510-bee2-628a20f0caad/main/0.log" Nov 03 16:11:32.447882 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:11:32.447787 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qqz7l_72f19618-4cf9-49cf-8416-187d0de0cf0b/main/0.log" Nov 03 16:11:38.701233 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:11:38.701188 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:12:06.852310 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:06.852273 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65f6786cb5-5wwgn_83bb0c49-dd3e-4323-b755-b78034bd8c30/manager/0.log" Nov 03 16:12:07.430155 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:07.430115 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65f6786cb5-5wwgn_83bb0c49-dd3e-4323-b755-b78034bd8c30/manager/0.log" Nov 03 16:12:08.310596 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:08.310568 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-f7dkj_0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8/payload-processing/0.log" Nov 03 16:12:08.558040 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:08.558007 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-f7dkj_0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8/payload-processing/0.log" Nov 03 16:12:08.704826 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:12:08.704686 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:12:18.843463 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:18.843418 4290 scope.go:122] "RemoveContainer" containerID="0d161defa6ea4aa989d6d75ac1d6a1b84ba6820ece14029d3324abd76b92d2c5" Nov 03 16:12:38.701884 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:12:38.701848 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:12:53.431497 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:53.431462 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65f6786cb5-5wwgn_83bb0c49-dd3e-4323-b755-b78034bd8c30/manager/0.log" Nov 03 16:12:54.040802 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:54.040768 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65f6786cb5-5wwgn_83bb0c49-dd3e-4323-b755-b78034bd8c30/manager/0.log" Nov 03 16:12:54.830818 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:54.830790 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-f7dkj_0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8/payload-processing/0.log" Nov 03 16:12:55.072016 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.071985 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-f7dkj_0cae1c25-ed2d-4b5a-b4dc-0f87dfc219e8/payload-processing/0.log" Nov 03 16:12:55.435245 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.435204 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-76mwr/must-gather-2492k"] Nov 03 16:12:55.435776 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.435755 4290 cpu_manager.go:414] "containerMap: removing container" podUID="b0d738eb-d32e-456c-b920-7b03b49d0af9" containerName="revoke-keys" Nov 03 16:12:55.435827 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.435775 4290 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d738eb-d32e-456c-b920-7b03b49d0af9" containerName="revoke-keys" Nov 03 16:12:55.435901 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.435892 4290 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d738eb-d32e-456c-b920-7b03b49d0af9" containerName="revoke-keys" Nov 03 16:12:55.439405 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.439390 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-76mwr/must-gather-2492k" Nov 03 16:12:55.481667 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.481635 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bgv\" (UniqueName: \"kubernetes.io/projected/cc87af35-eb6b-4f13-94e9-01abf8de5333-kube-api-access-f7bgv\") pod \"must-gather-2492k\" (UID: \"cc87af35-eb6b-4f13-94e9-01abf8de5333\") " pod="openshift-must-gather-76mwr/must-gather-2492k" Nov 03 16:12:55.481849 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.481680 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cc87af35-eb6b-4f13-94e9-01abf8de5333-must-gather-output\") pod \"must-gather-2492k\" (UID: \"cc87af35-eb6b-4f13-94e9-01abf8de5333\") " pod="openshift-must-gather-76mwr/must-gather-2492k" Nov 03 16:12:55.492304 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.492279 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-76mwr\"/\"kube-root-ca.crt\"" Nov 03 16:12:55.541927 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.541898 4290 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-76mwr\"/\"openshift-service-ca.crt\"" Nov 03 16:12:55.542067 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.542012 4290 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-76mwr\"/\"default-dockercfg-msnrf\"" Nov 03 16:12:55.545215 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.545194 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-76mwr/must-gather-2492k"] Nov 03 16:12:55.582193 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.582168 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bgv\" (UniqueName: \"kubernetes.io/projected/cc87af35-eb6b-4f13-94e9-01abf8de5333-kube-api-access-f7bgv\") pod \"must-gather-2492k\" (UID: \"cc87af35-eb6b-4f13-94e9-01abf8de5333\") " pod="openshift-must-gather-76mwr/must-gather-2492k" Nov 03 16:12:55.582363 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.582223 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cc87af35-eb6b-4f13-94e9-01abf8de5333-must-gather-output\") pod \"must-gather-2492k\" (UID: \"cc87af35-eb6b-4f13-94e9-01abf8de5333\") " pod="openshift-must-gather-76mwr/must-gather-2492k" Nov 03 16:12:55.582573 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.582555 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cc87af35-eb6b-4f13-94e9-01abf8de5333-must-gather-output\") pod \"must-gather-2492k\" (UID: \"cc87af35-eb6b-4f13-94e9-01abf8de5333\") " pod="openshift-must-gather-76mwr/must-gather-2492k" Nov 03 16:12:55.643606 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.643579 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bgv\" (UniqueName: \"kubernetes.io/projected/cc87af35-eb6b-4f13-94e9-01abf8de5333-kube-api-access-f7bgv\") pod \"must-gather-2492k\" (UID: \"cc87af35-eb6b-4f13-94e9-01abf8de5333\") " pod="openshift-must-gather-76mwr/must-gather-2492k" Nov 03 16:12:55.755154 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:55.755076 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-76mwr/must-gather-2492k" Nov 03 16:12:56.135015 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:56.134984 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-76mwr/must-gather-2492k"] Nov 03 16:12:56.135979 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:12:56.135952 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc87af35_eb6b_4f13_94e9_01abf8de5333.slice/crio-fe7531309732726c6aedfbecd8e2f84418a6555b2a2c7d85ae13118d40058b68 WatchSource:0}: Error finding container fe7531309732726c6aedfbecd8e2f84418a6555b2a2c7d85ae13118d40058b68: Status 404 returned error can't find the container with id fe7531309732726c6aedfbecd8e2f84418a6555b2a2c7d85ae13118d40058b68 Nov 03 16:12:56.138094 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:56.138074 4290 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:12:56.205167 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:56.205137 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76mwr/must-gather-2492k" event={"ID":"cc87af35-eb6b-4f13-94e9-01abf8de5333","Type":"ContainerStarted","Data":"fe7531309732726c6aedfbecd8e2f84418a6555b2a2c7d85ae13118d40058b68"} Nov 03 16:12:58.219657 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:58.219616 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76mwr/must-gather-2492k" event={"ID":"cc87af35-eb6b-4f13-94e9-01abf8de5333","Type":"ContainerStarted","Data":"4fc61d4650ce9451560c3470f235ef4688ac8cd2fa0844f3c1978b1093e97e74"} Nov 03 16:12:58.219657 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:58.219665 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76mwr/must-gather-2492k" event={"ID":"cc87af35-eb6b-4f13-94e9-01abf8de5333","Type":"ContainerStarted","Data":"9b053bfdcc58cc4c88737bae414de533d9494c6c37642f044c7024077829266d"} Nov 03 16:12:58.338545 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:12:58.338474 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-76mwr/must-gather-2492k" podStartSLOduration=2.164825927 podStartE2EDuration="3.338454064s" totalImagesPullingTime="1.173628137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:12:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:12:58.335880151 +0000 UTC m=+3885.766837957" watchObservedRunningTime="2026-11-03 16:12:58.338454064 +0000 UTC m=+3885.769411873" Nov 03 16:13:05.134095 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:05.134063 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hwchb_f997ae6c-4cdd-4def-ac45-e81c30896420/global-pull-secret-syncer/0.log" Nov 03 16:13:08.707853 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:08.707818 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q5rxt_c4f18ec9-6d01-4a3d-9845-58b607a6225f/konnectivity-agent/0.log" Nov 03 16:13:08.726354 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:13:08.726308 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:13:09.894393 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:09.894352 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-35.us-west-1.compute.internal_d955cef443a4983dada78f8e0a328c65/haproxy/0.log" Nov 03 16:13:13.557268 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:13.557234 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 16:13:13.561077 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:13.561055 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qpjj8_03a0fb03-c422-430a-aa4b-6226262bcb73/extract/0.log" Nov 03 16:13:13.561231 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:13.561153 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 16:13:13.565065 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:13.565042 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cq52k_f351bf29-892f-4c6f-9c1a-b284adb73e58/extract/0.log" Nov 03 16:13:14.656032 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.655981 4290 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv"] Nov 03 16:13:14.665433 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.665406 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.742620 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.742589 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv"] Nov 03 16:13:14.812032 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.811996 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e913dda-a6dc-4e6a-a59b-077b47590693-sys\") pod \"perf-node-gather-daemonset-hnrzv\" (UID: \"2e913dda-a6dc-4e6a-a59b-077b47590693\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.812234 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.812056 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e913dda-a6dc-4e6a-a59b-077b47590693-podres\") pod \"perf-node-gather-daemonset-hnrzv\" (UID: \"2e913dda-a6dc-4e6a-a59b-077b47590693\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.812234 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.812106 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qh5w\" (UniqueName: \"kubernetes.io/projected/2e913dda-a6dc-4e6a-a59b-077b47590693-kube-api-access-5qh5w\") pod \"perf-node-gather-daemonset-hnrzv\" (UID: \"2e913dda-a6dc-4e6a-a59b-077b47590693\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.812234 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.812218 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e913dda-a6dc-4e6a-a59b-077b47590693-proc\") pod \"perf-node-gather-daemonset-hnrzv\" (UID: \"2e913dda-a6dc-4e6a-a59b-077b47590693\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.812544 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.812303 4290 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e913dda-a6dc-4e6a-a59b-077b47590693-lib-modules\") pod \"perf-node-gather-daemonset-hnrzv\" (UID: \"2e913dda-a6dc-4e6a-a59b-077b47590693\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.913185 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.913051 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e913dda-a6dc-4e6a-a59b-077b47590693-sys\") pod \"perf-node-gather-daemonset-hnrzv\" (UID: \"2e913dda-a6dc-4e6a-a59b-077b47590693\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.913185 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.913168 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e913dda-a6dc-4e6a-a59b-077b47590693-podres\") pod \"perf-node-gather-daemonset-hnrzv\" (UID: \"2e913dda-a6dc-4e6a-a59b-077b47590693\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.913440 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.913225 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qh5w\" (UniqueName: \"kubernetes.io/projected/2e913dda-a6dc-4e6a-a59b-077b47590693-kube-api-access-5qh5w\") pod \"perf-node-gather-daemonset-hnrzv\" (UID: \"2e913dda-a6dc-4e6a-a59b-077b47590693\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.913440 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.913282 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e913dda-a6dc-4e6a-a59b-077b47590693-sys\") pod \"perf-node-gather-daemonset-hnrzv\" (UID: \"2e913dda-a6dc-4e6a-a59b-077b47590693\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.913440 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.913352 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e913dda-a6dc-4e6a-a59b-077b47590693-proc\") pod \"perf-node-gather-daemonset-hnrzv\" (UID: \"2e913dda-a6dc-4e6a-a59b-077b47590693\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.913440 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.913387 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e913dda-a6dc-4e6a-a59b-077b47590693-podres\") pod \"perf-node-gather-daemonset-hnrzv\" (UID: \"2e913dda-a6dc-4e6a-a59b-077b47590693\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.913440 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.913422 4290 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e913dda-a6dc-4e6a-a59b-077b47590693-lib-modules\") pod \"perf-node-gather-daemonset-hnrzv\" (UID: \"2e913dda-a6dc-4e6a-a59b-077b47590693\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.913812 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.913466 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e913dda-a6dc-4e6a-a59b-077b47590693-proc\") pod \"perf-node-gather-daemonset-hnrzv\" (UID: \"2e913dda-a6dc-4e6a-a59b-077b47590693\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.913812 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.913576 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e913dda-a6dc-4e6a-a59b-077b47590693-lib-modules\") pod \"perf-node-gather-daemonset-hnrzv\" (UID: \"2e913dda-a6dc-4e6a-a59b-077b47590693\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.979082 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.979049 4290 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qh5w\" (UniqueName: \"kubernetes.io/projected/2e913dda-a6dc-4e6a-a59b-077b47590693-kube-api-access-5qh5w\") pod \"perf-node-gather-daemonset-hnrzv\" (UID: \"2e913dda-a6dc-4e6a-a59b-077b47590693\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:14.989857 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:14.989828 4290 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:15.255850 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:15.255717 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b7867b55-9l2fl_8c0f5828-4470-45ae-af7b-3e2defa2bb3e/authorino/0.log" Nov 03 16:13:15.385634 ip-10-0-139-35 kubenswrapper[4290]: W1103 16:13:15.385599 4290 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2e913dda_a6dc_4e6a_a59b_077b47590693.slice/crio-03c8c60c6491128f5e0b928cc32dbfec58299d0f1817baaec17fecb03e8b25d9 WatchSource:0}: Error finding container 03c8c60c6491128f5e0b928cc32dbfec58299d0f1817baaec17fecb03e8b25d9: Status 404 returned error can't find the container with id 03c8c60c6491128f5e0b928cc32dbfec58299d0f1817baaec17fecb03e8b25d9 Nov 03 16:13:15.386336 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:15.386299 4290 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv"] Nov 03 16:13:15.407769 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:15.407738 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" event={"ID":"2e913dda-a6dc-4e6a-a59b-077b47590693","Type":"ContainerStarted","Data":"03c8c60c6491128f5e0b928cc32dbfec58299d0f1817baaec17fecb03e8b25d9"} Nov 03 16:13:15.592000 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:15.591927 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8tnvm_36653232-724e-4a78-b661-5a5c6adc541e/manager/0.log" Nov 03 16:13:16.417650 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:16.417611 4290 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" event={"ID":"2e913dda-a6dc-4e6a-a59b-077b47590693","Type":"ContainerStarted","Data":"b6e23ef0d4fc3f0bd064b51b75a9eda856d9210437c3f057c3fb4c12ca6b70f8"} Nov 03 16:13:16.418175 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:16.417676 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:16.539381 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:16.539325 4290 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" podStartSLOduration=2.539310081 podStartE2EDuration="2.539310081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:16.535778523 +0000 UTC m=+3903.966736330" watchObservedRunningTime="2026-11-03 16:13:16.539310081 +0000 UTC m=+3903.970267886" Nov 03 16:13:16.965842 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:16.965802 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-fzcg4_cdfe745b-0aad-43f7-aa54-1c44d45bde19/manager/0.log" Nov 03 16:13:18.870636 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:18.870605 4290 scope.go:122] "RemoveContainer" containerID="b8d534a887dfb1af873c473b164686d143d5a6d67cb870f1bd6ddcabdaf321a4" Nov 03 16:13:20.497058 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:20.497028 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ceb2931e-109a-4cec-ac2e-3b2f18e5de55/alertmanager/0.log" Nov 03 16:13:20.917376 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:20.917346 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ceb2931e-109a-4cec-ac2e-3b2f18e5de55/config-reloader/0.log" Nov 03 16:13:21.338721 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:21.338623 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ceb2931e-109a-4cec-ac2e-3b2f18e5de55/kube-rbac-proxy-web/0.log" Nov 03 16:13:21.760604 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:21.760570 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ceb2931e-109a-4cec-ac2e-3b2f18e5de55/kube-rbac-proxy/0.log" Nov 03 16:13:22.181350 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:22.181312 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ceb2931e-109a-4cec-ac2e-3b2f18e5de55/kube-rbac-proxy-metric/0.log" Nov 03 16:13:22.439225 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:22.439144 4290 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-hnrzv" Nov 03 16:13:22.605851 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:22.605820 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ceb2931e-109a-4cec-ac2e-3b2f18e5de55/prom-label-proxy/0.log" Nov 03 16:13:23.030874 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:23.030843 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ceb2931e-109a-4cec-ac2e-3b2f18e5de55/init-config-reloader/0.log" Nov 03 16:13:24.002057 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:24.002023 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hf224_85b5f3bf-df66-4584-8a1b-92610ae8d8dc/kube-state-metrics/0.log" Nov 03 16:13:24.416224 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:24.416187 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hf224_85b5f3bf-df66-4584-8a1b-92610ae8d8dc/kube-rbac-proxy-main/0.log" Nov 03 16:13:24.840495 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:24.840423 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hf224_85b5f3bf-df66-4584-8a1b-92610ae8d8dc/kube-rbac-proxy-self/0.log" Nov 03 16:13:25.268210 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:25.268182 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6699cbb5b6-xhr2d_2d45ed90-f053-4277-85a0-fde14f7ae154/metrics-server/0.log" Nov 03 16:13:25.695438 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:25.695405 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-8smb5_3ca19e3b-71f2-4067-a8ae-1a4b5afe14dd/monitoring-plugin/0.log" Nov 03 16:13:28.736383 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:28.736351 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9zb9h_c7f77efd-9feb-4054-848b-fc10fe76a318/node-exporter/0.log" Nov 03 16:13:29.171530 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:29.171495 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9zb9h_c7f77efd-9feb-4054-848b-fc10fe76a318/kube-rbac-proxy/0.log" Nov 03 16:13:29.609076 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:29.609003 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9zb9h_c7f77efd-9feb-4054-848b-fc10fe76a318/init-textfile/0.log" Nov 03 16:13:30.034844 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:30.034810 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6vg8h_dda38876-5e32-4413-bdfc-63feb921f660/kube-rbac-proxy-main/0.log" Nov 03 16:13:30.483274 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:30.483242 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6vg8h_dda38876-5e32-4413-bdfc-63feb921f660/kube-rbac-proxy-self/0.log" Nov 03 16:13:30.910823 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:30.910791 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6vg8h_dda38876-5e32-4413-bdfc-63feb921f660/openshift-state-metrics/0.log" Nov 03 16:13:34.430420 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:34.430390 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-pqswq_41e3b9c7-b796-4aab-8324-4c7a0a6c9429/prometheus-operator/0.log" Nov 03 16:13:34.848723 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:34.848624 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-pqswq_41e3b9c7-b796-4aab-8324-4c7a0a6c9429/kube-rbac-proxy/0.log" Nov 03 16:13:35.270939 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:35.270906 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-kfw9b_9846b382-908f-452b-afb6-083dfc331d62/prometheus-operator-admission-webhook/0.log" Nov 03 16:13:35.699211 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:35.699178 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f87df669-jfhpm_442059db-4817-43db-8096-2f9f0511c71c/telemeter-client/0.log" Nov 03 16:13:36.120856 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:36.120783 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f87df669-jfhpm_442059db-4817-43db-8096-2f9f0511c71c/reload/0.log" Nov 03 16:13:36.542014 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:36.541984 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f87df669-jfhpm_442059db-4817-43db-8096-2f9f0511c71c/kube-rbac-proxy/0.log" Nov 03 16:13:36.977078 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:36.977049 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bd85bcf57-n79t2_653f8239-2130-49de-9dd8-c45f1f161ee8/thanos-query/0.log" Nov 03 16:13:37.407492 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:37.407438 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bd85bcf57-n79t2_653f8239-2130-49de-9dd8-c45f1f161ee8/kube-rbac-proxy-web/0.log" Nov 03 16:13:37.832182 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:37.832104 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bd85bcf57-n79t2_653f8239-2130-49de-9dd8-c45f1f161ee8/kube-rbac-proxy/0.log" Nov 03 16:13:38.262977 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:38.262950 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bd85bcf57-n79t2_653f8239-2130-49de-9dd8-c45f1f161ee8/prom-label-proxy/0.log" Nov 03 16:13:38.509614 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:38.509566 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6644c85865t8j8h_f9ee4847-f5a9-4549-ba96-3e7a9b79db99/istio-proxy/0.log" Nov 03 16:13:38.683669 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:38.683637 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bd85bcf57-n79t2_653f8239-2130-49de-9dd8-c45f1f161ee8/kube-rbac-proxy-rules/0.log" Nov 03 16:13:38.703061 ip-10-0-139-35 kubenswrapper[4290]: E1103 16:13:38.703030 4290 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8qkv" containerName="extractor" Nov 03 16:13:39.110446 ip-10-0-139-35 kubenswrapper[4290]: I1103 16:13:39.110371 4290 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bd85bcf57-n79t2_653f8239-2130-49de-9dd8-c45f1f161ee8/kube-rbac-proxy-metrics/0.log"